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The Military Operations Research Society (MORS) 


The purpose of the Military Operations Research Society (MORS) is to enhance the 
quality and effectiveness of classified and unclassified military operations research. To 
accomplish this purpose, the Society provides media for professional exchange and peer 
criticism among students, theoreticians, practitioners and users of military operations 
research. These media consist primarily of the traditional annual MORS Symposia 
(classified), their published abstracts or proceedings, special mini-symposia, workshops, 
colloquia and special purpose monographs and other publications. MORS publishes two 
quarterly periodicals, PHALANX and Military Operations Research. PHALANX is the 
MORS bulletin and Military Operations Research is a referred journal. The forum 
provided by these media is directed to display the state of the art, to encourage consistent 
professional quality, to stimulate communication and interaction between practitioners 
and users, and to foster the interest and development of students of operations research. 
In performing its function, the Military Operations Research Society does not make or 
advocate official policy nor does it attempt to influence the formulation of policy. 
Matters discussed or statements made during the course of its symposia or printed in its 
publications represent the positions of the individual participants and authors and not of 
the Society. 

The Military Operations Research Society is operated by a Board of Directors consisting 
of 30 members, 28 of whom are elected by vote of the Board to serve a term of four 
years. The persons nominated for this election are normally individuals who have 
attained recognition and prominence in the field of military operations research and who 
have demonstrated an active interest in its programs and activities. The remaining two 
members of the Board of Directors are the Immediate Past President who serves by right 
and the Executive Vice President who serves as a consequence of his position. A limited 
number of Advisory Directors are appointed from time to time, usually for a one-year 
term, to perform some particular function. 

MORS is Sponsored by: 

• The Deputy Under Secretary of the Army (Operations Research) 

• The Director, Assessment Division, Office of the Chief of Naval Operations 

• The Director of Command and Control, Deputy Chief of Staff, Air and Space 
Operations, US Air Force 

• The Commanding General, Marine Corps Combat Development Command 

• The Director of Force Structure, Resources and Assessment, The Joint Staff 

• The Director Program Analysis and Evaluation, Office Secretary of Defense 
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Executive Summary 


Background 

Leaders in the DoD analysis community are 
calling for an examination of how we perform 
analyses. They are asking us whether our 
analysis practices warrant examination with a 
view towards improving their effectiveness. 

The management of knowledge, how to prepare 
to perform the next analysis, how to perform the 
next analysis, how to gather information needed 
to support analysis, and how to find and apply 
analytic insights from past studies, are becoming 
principal factors in being able to meet decision 
makers’ expectations in regard to both the 
quality of an analysis and the speed with which 
an analysis is accomplished. 

"Making Analysis Relevant" was the topic of a 
PHALANX article by Vince Roske in early 
1998.' Mr Roske reiterated a recipe provided by 
John D. (Dave) Robinson, MG USA, Ret, when 
he was the Director of J-8. The recipe is 
provided below: 

• What’s the question? 

• What’s the “real” question? 

• What do the final slides look like? 

• What do / already know ? 

• How do I get the remaining information that 
1 need? 

Dave Robinson’s recipe is reminiscent of the 
practical battle drills used by the military to train 
soldiers to do basic tasks. It is simple, it is 
memorable, and it will get the job done. Mr 
Roske goes on to look deeper at each of the 
recipe steps and discusses the issues associated 
with performing analysis today. He explains 
that we must focus on the purpose of the 
analysis and not become wed to models that 
don’t fit the analysis. Furthermore, he explains 
what the analyst must do to keep the analysis 
relevant to the decision maker. 


Discussion 

The keys (and related impediments) to effective 
analysis in DoD are; 

The analysts, 

• Turnover and reduction in numbers of 
analyst positions in DoD organizations is 
creating a critical situation. 

The knowledge that analysts access, 

• DoD analysts are spread around the world 
and find it very difficult to collaborate. 

The process by which analysts perform their 
work, 

• DoD does not have a best practices guide to 
aid analysts in their work. 

The tools that analysts employ to perform their 
work, 

• The analytical methods and tools employed 
by DoD analysts are increasingly complex 
as are the problems to which the methods 
and tools are being applied. 

The effectiveness with which analysts 
communicate analytical results to decision 
makers, 

• The quantity, variety and complexity of 
military operations is increasing — the time 
to perform analysis and communicate results 
to decision makers is decreasing. 

The proper management of knowledge can 
improve analytical efficiency by providing a 
framework, tools and techniques to reuse 
intellectual assets. By marshalling resources to 
respond to opportunities and threats, our 
responsiveness could be vastly improved. And 
by bringing people together across time and 
geography to share ideas and insight, intellectual 
innovation can flourish and bear richer insight 
for decision makers. 


Mr Vincent P. Roske, Jr., “Making Analysis Relevant,' 
PHALAWX Volume 31, Number 1, March 1998 
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This special meeting, comprising analysts and 
subject matter experts in approaches to analysis, 
concentrated on the improvement of the analysis 
practice throughout DoD. To achieve this end, 
participants enumerated potential issues related 
to a broad move towards more collaborative 
analysis and enterprise knowledge management 
and define the appropriate policies, 
organizational structure, processes, technologies 
and training needed to institutionalize an 
improved practice of analysis in DoD. These 
issues were addressed by Working Groups 
(WGs), addressing the spectrum of dimensions 
of organizational change, as well as supporting 
infrastructures and supporting tools and 
techniques. 

Goals and Objectives 

The body of analytical methods, techniques and 
tools and how we apply them was the focus of 
this workshop. We examined how we carry out 
our professional work and the need to evolve. 
Questions addressed included: 

• What should be the customary manner for 
performing defense-related analyses? 

• How do we manage the knowledge we 
derive from our analyses? 

• How do we support both fast turn-around 
and more deliberate analyses? 

• What problems are not well served by the 
current state of the practice? 

• What innovative developments in analysis 
tools and methods, including Knowledge 
Management, hold potential value for DoD 
analysis? 

Goals 

This workshop afforded the military operations 
research community an opportunity to achieve 
the following goals: 


• A vision for improving analytic practice and 
the management of supporting analytic 
knowledge in DoD. 

• A plan to realize the vision to improve 
analytic practice and the management of 
supporting analytic knowledge in DoD. 

• A definition of how Military Operations 
Research Society (MORS) may work with 
the Sponsors over time to implement 
improvements to analytic practice and to the 
management of analytic knowledge in DoD. 

Objectives 

The objectives were to: 

• Define a desired analytic practice approach 
in DoD in terms of policy, organization, 
process, technology and training. 

• Define current analytic practice in DoD in 
terms of policy, organization, process, 
technology and training. 

• Identify weaknesses in the current practice 
relative to current and future requirements. 

• Produce an actionable plan to move DoD to 
the desired state. 

• Define mechanisms to be established within 
MORS to work with the DoD Sponsors to 
execute the plan. 

Approach 

In order to achieve these goals and objectives, 
the subject of analytic practice was examined in 
a number of different contexts. Working groups 
examined analytic practice within each 
contextual framework. WGs were asked to 
address some baseline issues (as a starting point) 
and provide specific results. The workshop 
sessions are listed in Table E-1. 




A shared understanding of the need to 
evolve the practice of military operations 
analysis in DoD. 
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Sessions 

Topics 

Leadership 

Keynote 

Session 

Dr Ruth David: 

Knowledge 

Management 

Jim Johnson: 
Perspectives on 
Requirements 
Analysis 

Bruce Powers: 
Perspectives on 
Capabilities 
Analysis 

Workshop Lead 
Proponent: 

Vince Roske 

Workshop Chair: 
Denis Clements 

Workshop Co- 
Chair: Sue 

Iwanski 

Plenary 

Sessions 

Mike Yoemans: 
Enterprise 
Knowledge 
Management 

Dr Tom Allen: 

Joint Analysis 

QDR 2001 and 
Beyond 

Working 

Groups 

Management of 
Analytic 

Knowledge 

WG Chair: Mike 
Yoemans 

WG Co-Chair: 

Gary Coe 


Enabling 

Collaboration 

WG Chair: COL 
Bob Clemence 

WG Co-Chair: 

Lt Col Kirk Yost 


New Analysis 
Challenges 

WG Chair: Dr 
Steve Pilnick 

WG Co-Chair: 

Lt Col Suzanne 
Beers 


New Analysis 

Tools and 

Methods 

WG Chair: LTC 
Dan Maxwell 

WG Co-Chairs: 
John Furman and 
Dr A1 Brandstein 

Synthesis 

Group 

Consolidating 
Recommendations 
and Developing 
an Action Plan 

Group Chair: Dr 
Stuart Starr, FS 
Group Co- 
Chairs: Dr Jerry 
Kotchka, Dr 

Tom Allen 


Table E-1. Workshop Organization 


Working Group Sessions 

The workshop attendees were organized into 
four subject area WGs of about 20 to 30 
members each plus a Synthesis Group to address 
the overall goals and objectives. 


WG 1, Management of Analytic Knowledge, 
examined the manner in which analytic 
knowledge is discovered, expanded, organized 
and shared. 

WG 2, Enabling Collaboration, examined the 
process, tools and techniques that support 
distributed analytic teams in performing analytic 
tasks. 

WG 3, New Analysis Challenges, examined 
military operations analysis questions of interest 
for which the established military operations 
analysis community apparently does not have 
answers. 

WG 4, New Analysis Tools and Methods, 
examined emerging developments in analytical 
approaches to complex systems to assess 
potential applications to challenging military 
operations analysis problems. 

The Synthesis Group sought to discover the 
common themes that tie together the efforts of 
the four subject area WGs and provide feedback 
to the groups on a continuing basis. 

Workshop Synopsis 

Figure E-1 shows an illustration of an ideal end- 
state for the current efforts to improve analytical 
practices. There are three families of models and 
simulations: M&S embedded in real-time 
command and control systems; training and 
exercise support simulations; and models and 
simulations used to support analysis (to include 
Test and Evaluation (T&E) analysis and analysis 
of non-warfighting issues such as infrastructure 
and business process improvement). Within each 
domain are a set of models and simulations 
tailored to the purpose at hand. No single model 
will apply universally across all domains, but in 
this ideal end state we have data, software 
objects and process representations shared 
across the domains where it is logically 
consistent with the model and purpose. 
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Figure E-1. Potential Future “Framework” 

Infrastructure programs currently under 
development are assumed to be in place in this 
end state, to include the High-Level Architecture 
(HLA) for simulations and the DII COE linking 
all areas together. HLA federations, which may 
include simulations widely, distributed over 
space, will exist where appropriate, but these 
federations should be standardized and validated 
for consistency across federates before use. The 
creation of “on the fly” federations that grab 
objects and federates scattered across the DII to 
create unique environments is probably still not 
possible in this end state because there is no 
guarantee that the representations will be 
internally consistent (consistency is not assured 
even if data is passed through HLA), and it is 
unlikely that a single analyst or team located at a 
single site will be familiar with all of the objects 
or federates potentially available. Instead, these 
simulations may be combined through the use of 
collaborative analysis rather than software 
linkages. 

All of the tools are presumed to be in a bed of 
warm data. That is, there is a data infrastructure 
that makes data readily available (with 
exhaustive metadata explaining the derivation 
and accredited use of the data) for all of these 
systems. For example, an analysis of real-world 
capability should be able to draw on the same 
data set showing current status and deployments 
of units that a real-time command and control 
system might use. Similarly, an analysis of a 
future capability (for example, a 2010 force in a 


particular scenario) should be able to draw on a 
validated and commonly used set of data to 
represent the scenario, system performance, 
OPFOR performance, etc. Generation and 
storage of this data requires a significant 
resource commitment within DoD, but it is 
necessary to achieve the desired end state. 

A key feature of this environment is that the 
output of the models, as well as the results of 
specific analyses and the lessons learned from 
training and exercises are also added to the 
common database. This provides a useful 
source of information that can be of particular 
help to quick turn-around analyses, which may 
synthesize the outputs of previous models, 
exercises and analyses. 

The Action Plan 

The Synthesis Group identified a number of 
actions that should become part of a long-range 
action plan that the community should consider 
and flesh out over the next few years as it 
evolves the practice of military analysis in DoD. 

There are several steps that MORS can 
implement immediately that have little work 
impact but enormous potential to improve the 
ability of this community to connect with each 
other and better meet the over all needs of its 
membership. In addition to the items listed 
below, the working groups provided a number of 
general and specific steps that will help the 
community learn to harness knowledge 
management and collaboration, as well as to 
address emerging issues in the changing security 
environment. Generalized recommendations 
endorsed by the workshop include working the 
people, process and technology problems in 
order to ensure a better final product. 

Actions for MORS include: 

• Near Term: 

Update the MORS membership directory 
and database with more information on each 
member. 

Evolve the role of the Electronic Media 
Committee to address enterprise knowledge 
management. 
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Contribute to a Code of Best Practices for 
Analysis and Joint Experimentation. 

• Mid Term: 

Undertake focused workshop activity on this 
topic. 

• Far Term: 

Use the Education Colloquia to address 
ideas that Knowledge Management is an 
important element of professional 
development. 

Actions for DoD include: 

The working group recommendations 
provide a phased approach to knowledge 
management and collaboration 
implementation by focusing on people, 
process, and technology and product, as well 
as list challenges and promising tools and 
methodologies to meet those challenges. 

Table E-2 summarizes the focus and potential 
impact of the specific actions recommended. 
First, cultural changes are a necessary 
prerequisite to effective transformation of the 
analysis community (e.g., the need to dispel the 
myth that “knowledge is power”). Second are 
several organizational issues that must be 
addressed. One of these is to ensure top 
management buys in to this transformation. 
Another includes reassessing how security issues 
can be addressed in an information-centric, open 
environment. In the third area, decision-makers 
must recognize that the analysts involved in all 
phases of the assessment process constitute a 
critical intellectual resource. 


_ Summary Recommendations _ 

Vision/Policy: Develop and promulgate an Enterprise 

Vision and strategic directions. __ 

Culture: Take action to remove deeply rooted 
cultural barriers (e.g., remove the incentives to hoard 

knowledge) _ 

Organization: Ensure that there is buy-in by Top 
Management. Investigate organizational constructs 
including business rules that best enable global 

collaboration of military analysts. __ 

People: Provide incentives to analysts to collaborate 

and share knowledge. __ 

Process: Institutionalize processes to enhance 
collaboration and the sharing of tacit and explicit 

knowledge among the analytical community. _ 

Analytic Tools: 

• Develop and emphasize joint standards in 
algorithm and data development. 

• Expand the development and maintenance of 
data warehouses. 

• Focus basic research and tool development to 

_ meet emerging challenges _ 

Table E-2. Summary Recommendations 

They must be given incentives to stimulate them 
to contribute and draw from future knowledge 
warehouses. Tying all this together are the links 
between the vision and the high level guidance 
to the analysis process and the primary products 
of that process (i.e., effective, timely, rigorous 
analyses). To implement this linkage, a set of 
enhanced processes is needed (e.g., 
institutionalized processes to diffuse tacit and 
explicit knowledge) supported by a broad set of 
tools (e.g., knowledge warehouses, collaboration 
tools). These processes and tools are constrained 
by available resources (e.g., funds, manpower) 
such that a special funding mechanism may be 
required to satisfy time-urgent joint needs. In 
particular, basic research must be continued in 
those areas where understanding of 
phenomenology is not available (human decision 
processes. Information Operations/Information 
Warfare (lO/IW), etc) as well as the continuing 
development of analysis tools to address the 
emerging challenges. 
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The Synthesis Group feels that by addressing the 
needs of the community to implement our 
vision, the practice of analysis can evolve from 
its current position of a limited toolkit and 
methodologies to address a limited number of 
issues to a complete tool set connecting analysts 
and decision makers to address the full array of 
national security challenges. 

Summary 

This workshop has demonstrated that there is 
great potential to be realized from the harnessing 
of the knowledge in our heads and in the 
systems that support our analytical efforts. In 
addition, there is great power in undertaking a 
more collaborative approach to the analyses that 
we perform. The combination of enterprise 
knowledge management and collaboration can 
deliver better analysis products faster. 


The way ahead to realize this vision will be 
difficult. Nothing less than a cultural change is 
required. The demands of the national security 
environment require that we find ways to alter 
the information security environment to permit 
knowledge sharing and collaboration on an 
enterprise scale. Getting analysts to collaborate 
and breaking down the barriers to that 
collaboration and knowledge sharing are 
fundamental changes that require bottom up as 
well as top down change. 

The members of this workshop acknowledge the 
potential value of such change. It is now up to 
us to obtain the buy-in of the DoD leadership by 
convincing them that the benefits are worth the 
investment of time, money and energy to realize 
the vision articulated by this workshop. 
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Workshop Report 

Evolving the Practice of Military Operations Analysis in DoD 
Applying Knowledge Management 


Background 

Leaders in the DoD analysis community are 
calling for an examination of how we perform 
analyses. They are asking us whether our 
analysis practices warrant examination with a 
view towards improving their effectiveness. 

The management of knowledge, how to prepare 
to perform the next analysis, how to perform the 
next analysis, how to gather information needed 
to support analysis, and how to find and apply 
analytic insights from past studies, are becoming 
principal factors in being able to meet decision 
makers’ expectations in regard to both the 
quality of an analysis and the speed with which 
an analysis is accomplished. 

"Making Analysis Relevant" was the topic of a 
PHALANX article by Mr Vince Roske in early 
1998.^ Mr Roske reiterated a recipe provided by 
John D. (Dave) Robinson, MG USA, Ret, when 
he was the Director of J-8. The recipe is 
provided below: 

• What’s the question? 

• What’s the “real” question? 

• What do the final slides look like? 

• What do I already know ? 

• How do I get the remaining information that 
I need? 

Dave Robinson’s recipe is reminiscent of the 
practical battle drills used by the military to train 
soldiers to do basic tasks. It is simple, it is 
memorable and it will get the job done. Mr 
Roske goes on to look deeper at each of the 
recipe steps and discusses the issues associated 
with performing analysis today. He explains 
that we must focus on the purpose of the 
analysis and not become wed to models that 


don’t fit the analysis. Furthermore, he explains 
what the analyst must do to keep the analysis 
relevant to the decision maker. 

Discussion 

The first three steps in General Robinson’s 
recipe address critical steps in organizing and 
focusing an analysis or study. Answering 
“What’s the question?” and “What’s the “real” 
question?” enables the analyst to frame a study 
so that there is a chance that it may ultimately be 
relevant to the needs of the decision maker. 
Answering “What do the final slides look like?” 
up front serves the dual purpose of stressing the 
importance of communicating results to the 
decision maker and focusing the analysis efforts 
from the beginning. 

But the substantive content of the analysis is 
contained in the last two recipe steps: “What do I 
already know?” and “How do I get the 
remaining information I need?” Evolving the 
practice of military operations analysis in DoD 
must therefore substantively address these last 
two questions. 

One of the keys to successfully evolving the 
practice of analysis is the innovative 
management and application of knowledge, both 
tacit (the knowledge in the minds of the experts) 
and explicit (the knowledge that is recorded in 
various ways). There is an explosion of new 
thinking and new supporting technology to 
capture and apply knowledge more effectively. 
This area includes innovation in the discovery, 
expansion, capture, organization and sharing of 
knowledge. Innovations include: correlation 
landscapes, data mining tools, virtual 
information centers and influence diagrams. 


^ Mr Vincent P. Roske, Jr., “Making Analysis Relevant,” 
PHAMA/X Volume 31, Number 1, March 1998 
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The keys (and related impediments) to effective 
analysis in DoD are: 

The analysts, 

• Turnover and reduction in numbers of 
analyst positions in DoD organizations is 
creating a critical situation. 

The knowledge that analysts access, 

• DoD analysts are spread around the world 
and find it very difficult to collaborate. 

The process by which analysts perform their 
work, 

• DoD does not have a best practices guide to 
aid analysts in their work. 

The tools that analysts employ to perform their 
work, 

• The analytical methods and tools employed 
by DoD analysts are increasingly complex 
as are the problems to which the methods 
and tools are being applied. 

The effectiveness with which analysts 
communicate analytical results to decision 
makers, 

• The quantity, variety and complexity of 
military operations is increasing — the time 
to perform analysis and communicate results 
to decision makers is decreasing. 

Information is a source for good decision 
making, however, a piece of information is more 
valuable when presented in context, that is, with 
other pieces of related information that makes 
the original information more meaningful and 
actionable. For instance, a study of the 
effectiveness of a proposed Intelligence, 
Surveillance and Reconnaissance (ISR) system 
may provide important information, but it is 
much more meaningful and actionable when 
analyzed in the context of national security 
policy and DoD objectives and when related to 
operational outcomes. 

In DoD, we analysts don’t know what we know, 
and consequently, we are bound to reinvent the 
wheel many times. We do not have automated 
enterprise repositories which hold accessible. 


important, explicit knowledge nor do we have 
the means to locate the right subject matter 
experts quickly and easily to access their tacit 
knowledge. 

The proper management of knowledge can 
improve analytical efficiency by providing a 
framework, tools and techniques to reuse 
intellectual assets. By marshalling resources to 
respond to opportunities and threats, our 
responsiveness could be vastly improved. And 
by bringing people together across time and 
geography to share ideas and insight, intellectual 
innovation can flourish and bear richer insight 
for decision makers. 

This special meeting, comprising analysts and 
Subject Matter Experts (SMEs) in approaches to 
analysis, concentrated on the improvement of 
the analysis practice throughout DoD. To 
achieve this end, participants enumerated 
potential issues related to a broad move towards 
more collaborative analysis and enterprise 
knowledge management and define the 
appropriate policies, organizational structure, 
processes, technologies and training needed to 
institutionalize an improved practice of analysis 
in DoD. These issues were addressed by 
working groups, addressing the spectrum of 
dimensions of organizational change, as well as 
supporting infrastructures and supporting tools 
and techniques. 

Goals and Objectives 

The body of analytical methods, techniques and 
tools and how we apply them was the focus of 
this workshop. We examined how we carry out 
our professional work and the need to evolve. 

Questions addressed included: 

• What should be the customary manner for 
performing defense-related analyses? 

• How do we manage the knowledge we 
derive from our analyses? 

• How do we support both fast tum-around 
and more deliberate analyses? 
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• What problems are not well served by the 
current state of the practice? 

• What innovative developments in analysis 
tools and methods, including knowledge 
management, hold potential value for DoD 
analysis? 

Goals 

This workshop afforded the military operations 

research community an opportunity to achieve 

the following goals: 

• A shared understanding of the need to 
evolve the practice of military operations 
analysis in DoD. 

• A vision for improving analytic practice and 
the management of supporting analytic 
knowledge in DoD. 

• A plan to realize the vision to improve 
analytic practice and the management of 
supporting analytic knowledge in DoD. 

• A definition of how Military Operations 
Research Society (MORS) may work with 
the Sponsors over time to implement 
improvements to analytic practice and to the 
management of analytic knowledge in DoD. 

Objectives 

The objectives were to: 

• Define a desired analytic practice approach 
in DoD in terms of policy, organization, 
process, technology and training. 

• Define current analytic practice in DoD in 
terms of policy, organization, process, 
technology and training. 

• Identify weaknesses in the current practice 
relative to current and future requirements. 

• Produce an actionable plan to move DoD to 
the desired state. 

• Define mechanisms to be established within 
MORS to work with the DoD Sponsors to 
execute the plan. 


Approach 

In order to achieve these goals and objectives, 
the subject of analytic practice was examined in 
a number of different contexts. Working groups 
examined analytic practice within each 
contextual framework. WGs were asked to 
address some baseline issues (as a starting point) 
and provide specific results. The workshop 
sessions are listed in Table 1. 


Sessions 

Topics 

Leadership 

Keynote 

Session 

Dr Ruth David: 

Knowledge 

Management 

Jim Johnson: 
Perspectives on 
Requirements 
Analysis 

Bruce Powers: 
Perspectives on 
Capabilities 

Analysis 

Workshop Lead 
Proponent: Vince 
Roske 

Workshop Chair: 
Denis Clements 

Workshop Co- 
Chair: Sue 

Iwanski 

Plenary 

Sessions 

Mike Yoemans: 
Enterprise 
Knowledge 
Management 

Dr Tom Allen: 

Joint Analysis 

QDR 2001 and 
Beyond 

Working 

Groups 

Management of 
Analytic 

Knowledge 

WG Chair: Mike 
Yoemans 

WG Co-Chair: 

Gary Coe 


Enabling 

Collaboration 

WG Chair: COL 
Bob Clemence 

WG Co-Chair: Lt 
Col Kirk Yost 


New Analysis 
Challenges 

WG Chair: Dr 

Steve Pilnick 

WG Co-Chair: Lt 
Col Suzanne 

Beers 


New Analysis 

Tools and Methods 

WG Chair: LTC 
Dan Maxwell 

WG Co-Chairs: 
John Furman and 

Dr A1 Brandstein 

Synthesis 

Group 

Consolidating 
Recommendations 
and Developing an 
Action Plan 

Group Chair: Dr 
Stuart Starr, FS 
Group Co-Chairs: 
Dr Jerry Kotchka, 
Dr Tom Allen 


Table 1. Workshop Organization 
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Plenary Session Presentations 

The Keynote speaker was Dr Ruth David, CEO 
of ANSER a Federally Funded Research and 
Development Center (FFRDC), who discussed 
“Enterprise Knowledge Management.” In the 
context of a hypothetical story with the entire 
national security apparatus as the enterprise and 
Homeland Defense as the strategic objective, she 
explained that in the 2E' century our nation’s 
security would depend on effective knowledge 
management. Knowledge management was 
defined based on the article “On Knowledge 
Management” by Ramon Barquin in the 
November/December 1999 issue of the E-Gov 
Journal. “Knowledge Management is the 
process through which an enterprise uses its 
collective intelligence to accomplish its strategic 
objectives.” 

Dr David quoted Bran Ferren, President of Walt 
Disney Imagineering who stated that no “big 
idea” would survive a requirements process. 

She asserted that Enterprise Knowledge 
Management is definitely a big idea, which 
requires the reinvention of the way we operate 
and that our requirements’ processes and 
Program Objective Memorandum (POM) build 
cycles will not effectively support this. To 
successfully achieve this we need to focus on the 
four key elements critical to the success of 
Knowledge Management: Strategy, People, 
Process and Technology. 

The strategy provides the motivation, or sense of 
purpose for the activity. Effective knowledge 
management requires a shared goal or objective 
— and the strategy describes the way that goal 
will be achieved. People are the next critical 
element. Without people who create and use the 
knowledge, the whole issue is moot. People, or 
participants in the enterprise, must be motivated 
to share not only information, but knowledge as 
well. That is, they must do more than simply 
populate databases and other static repositories. 
They must actively and personally engage in 
building and using the collective intelligence of 
the enterprise. And, these participants must span 
the spectrum from those who collect and analyze 
information to those who take action based on 
the results. That is, the traditional “customers” 


of the analytic community — whether they are 
warfighters, policy makers, law enforcement 
officials or local first responders — must be 
active participants, expanding the collective 
intelligence of the enterprise. 

In terms of process, if Knowledge Management 
is defined as the overall process, then critical 
sub-processes include cooperation, 
communication and collaboration. Michael 
Schrage in Shared Minds: The New 
Technologies of Collaboration stated that the 
core processes must include robust cooperation, 
communication and collaboration. In his book, 
Mr Schrage defines cooperation as “people 
working together for a common purpose without 
necessarily having defined the purpose,” 
communication as “the exchange of 
information” and collaboration as “an act of 
shared creation and/or shared discovery.” 

Finally, we must have the enabling technologies 
to make connections, from people to machines, 
between machines and among people. 
Computation can be quite powerful, but it is 
fundamentally stupid! Emerging adaptive 
algorithms hold the potential for moving from 
“what you ask is what you get” to “what you 
need is what you get” in searching for 
information. This type of smart search has the 
potential to save significant time and effort. 

Mr Jim Johnson, Deputy Director for Theater 
Assessments and Planning, OSD, ODPA&E, 
discussed “Perspectives on Requirements 
Analysis.” He emphasized the need to do a 
thorough job of analyzing the trade space to 
understand not only cost versus effectiveness but 
also the relationship of cost and risk. 

Jim illustrated that the shape of the cost versus 
effectiveness curve in systems effectiveness 
analysis, when explored and well understood 
will reveal where this relationship may become 
ver>' important to the analysis. For example, if 
maximum effectiveness as a function of cost is 
achieved very early in the system effectiveness 
curve and the application of added dollars has 
little effect in growing effectiveness, one may 
then focus on that part of the curve where dollars 
buy significant value. 
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Also illustrated was the notion of buying down 
risk. As in the system effectiveness case, risk 
may be related to investment. Investments in 
force structure are made in light of anticipated 
threats. US force structure reduces the threat of 
war or diminishes the risk of unacceptable losses 
if military operations are undertaken. Again, 
one must explore the analysis space to 
understand the shape of the curve or the surface. 
This will allow the study to identify the region 
where added investment ceases to diminish risk 
in a meaningful way. 


To make results relevant he provided the 
following recipe aligned with Dave Robinson’s 
questions: 


The Question 

The Answer 

What is the question? 

Keep it simple. 

What is the real question? 

Don’t forget what you 
know. 

What do the final slides 
look like? 

Encapsulate complex 
representations. 

What do I already know? 

Know your audience. 

How do I get the 
remaining information I 
need? 

Be relevant. 


Table 2. Making Results Relevant 


Mr Bruce Powers, Director for Requirements 
Analysis, CNO Staff, N81, provided 
"Perspectives on Capabilities Analysis." He 
described the IWAPi/CPAM process that the 
Navy uses to plan the way ahead so that 
decisions made today do not foreclose flexibility 
in the future. 

Bruce identified the four primary components of 
the mission of the Deputy CNO for Resources, 
Warfare Requirements and Assessments: 

• Analyze uses of the Navy, including its 
goals in operations, providing insight to 
CNO, SECNAV and DoD. 

• Identify programmatic paths to provide 
required future capabilities. 

• Clarify consequences of each path, and 
possible blockages (e.g., funding and 
technology). 


• Keep at least one path open toward each 
operational goal. 

A very interesting feature of the IWAR/CPAM 
strategy is to ensure that total lifecycle cost for 
systems and follow-on systems is accounted for. 
The Navy employs a far-reaching planning 
horizon to ensure that the Navy: “Develops 
alternative force structure paths at the platform 
level for ships, submarines and aircraft that meet 
fiscal constraints. Each path will attempt to 
maximize warfighting capabilities while 
balancing recapitalization, modernization and 
operating costs.’’ 

Bruce summarized by stating that in building the 
future Navy there is a need to assess what the 
Navy will be doing — how to make it the most 
capable. He emphasized that this must be done 
within projected cost constraints (e.g., zero 
growth) and in light of other considerations such 
as maintaining program stability, a strong 
industrial base, etc. 

Mr Mike Yoemans, Chief Knowledge Officer, 
OSD, OASD (C3I), presented an overview of 
knowledge management during lunch on the 
first day of the workshop. 

Mike described a framework for managing 
knowledge. Figure 1 illustrates this framework. 



Figure 1. Epistemological Architecture 

He emphasized the need to focus on how 
knowledge is constructed through a bottom-up 
process that accumulates facts and meanings in 
order to develop data. Knowledge is derived by 
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human beings who combine data w'ith context. 
This may be understood by using the metaphor 
of a message. The message provides context 
and the data provides the combination of facts 
and meaning. Humans who take advantage of 
the knowledge they are able to accumulate have 
wisdom if they apply this knowledge in 
meaningful ways. This is a rather simple 
epistemology, however, it does provide a useful 
set of definitions and semantic relationships for 
discussions of Knowledge Management. 

Mike went on to emphasize the need for a 
Global Information Grid (GIG) that currently 
does not exist as a necessity for achieving 
information superiority. One of the goals of 
ASD (C3I) is to promote the development of 
Knowledge Management and a skill-based 
workforce throughout DoD. A trusting 
environment where there is a need and the 
incentive to share information is essential to the 
success of knowledge management. To counter 
the popular saying that knowledge is power, he 
quoted ADM Harold Gehman. "Knowledge, 
jealously guarded by an individual, is not power. 
Quite the opposite, it detracts from the power 
and effectiveness of the entire organization." 
"We must adapt to new information flow. 
Everyone must “know” what everyone else 
knows." 

Working Group Sessions 

The workshop attendees were organized into 
four subject area WGs of about 20 to 30 
members each plus a Synthesis Group to address 
the overall goals and objectives. 

WG 1, Management of Analytic Knowledge, 
examined the manner in which analytic 
knowledge is discovered, expanded, organized 
and shared. 

WG 2, Enabling Collaboration, examined the 
process, tools and techniques that support 
distributed analytic teams in performing analytic 
tasks. 

WG 3, New Analysis Challenges, examined 
military operations analysis questions of interest 
for which the established military operations 


analysis community apparently does not have 
answers. 

WG 4, New Analysis Tools and Methods, 
examined emerging developments in analytical 
approaches to complex systems to assess 
potential applications to challenging military 
operations analysis problems. 

The Synthesis Group sought to discover the 
common themes that tie together the efforts of 
the four subject area WGs and provide feedback 
to the groups on a continuing basis. 

In the following sections, the efforts and 
products of each of these five groups will be 
described in greater detail, sessions will be 
summarized and recommendations will be 
enumerated. 

Working Group 1 — Management of Analytic 
Knowledge 

Vision Statement: Informed decisions resulting 
from effective and efficient analyses enabled by 
the sharing, collaboration and creation of 
knowledge. 

WG 1 examined four questions; 

• How is analytic knowledge currently 
discovered, captured, expanded, organized 
and shared within the DoD analysis 
community? 

• What Knowledge Management techniques 
can be applied to improve DoD analysis? 

• What analytic practices (fast turn-around, 
deliberate, etc.) would make use of analytic 
knowledge? 

• What recommendations do you have to 
improve the existing analytic practice using 
knowledge management? 

This WG examined the literature that is defining 
the field of Knowledge Management. The group 
examined the manner in which analytic 
knowledge is discovered, expanded, organized 
and shared. It also addressed intelligent search 
engine technology and intelligent agent 
technology and how these techniques leverage 
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web-based stored data to meet real-time analysis 
requirements. 

WG1 Session Summary 

Session 1 - Introduction to Topic; The Analytic 
Enterprise 

Session 2 - Case Study on the Mobility 
Requirements Study; A Portal to Analysis 
Knowledge 

Session 3 - A Conceptual Framework for Using 
Knowledge Management; Case Study 
Knowledge Management Lessons from the 
Synthetic Theater of War 

Session 4 - DoD Practices for Acquiring and 
Storing Analytic Knowledge 

Session 5 - Working Discussion 

Session 6 - Working Discussion 

Moving From “Siloed” to Enterprise 
Knowledge 

The WG identified features of DoD 
organizations involved in analysis today, 
assessing that analytic knowledge tends to be 
protected as if held in a silo, accessible only to 
the organization holding the knowledge. 


“Siloed” Knowledge 

Enterprise Knowledge 

Information “silos” 
determined by organizational 
boundaries, divisions, 
functions, etc. 

Enterprise-wide integration 
of information resources. 

Study centered 

Issues centered - 
Information, people, tools, 
etc. 

Navigation via operating and 
file systems 

Navigation via hypertext 
browser links and search 
engines. 

Reliant on internally 
generated information. 

Integrates external 
information from strategic 
partners, suppliers, 
customers and third party 
sources. 

Information shared via e- 
mail. 

Information shared via 
knowledge portals. 


Table 3. Contrasting Views of Knowledge Access 


In contrast to this current state of "Siloed" 
Knowledge, the WG defined corresponding 
features of the desired state of enterprise 


knowledge where organizations can openly 
access knowledge from the entire analytic 
enterprise. 

Lessons from the Private Sector 

A study recently conducted by the Working 
Council for Chief Information Officers 
demonstrates that there are approaches already 
implemented in industry from which DoD may 
benefit. It should be emphasized that these are 
proven technologies and techniques. In fact, 
some of these technologies and techniques are 
either prototyped or in place in DoD. For 
example, both ASD(C3I) and US Joint Forces 
Command have implemented policies, 
techniques and technologies that are adapted 
from industry. These include: 

Thesaural Browsers — These browsers generate 
lists of related terms and resources allowing 
users to take advantage of otherwise unexpected 
associations between seemingly unrelated 
information sources and types. 

Personalized Intranet Portals — The creation of 
customizable intranet interfaces allows analysts 
selective access to information resources 
required to perform their jobs, enables analyst 
automation of simple, routine tasks and creates a 
vehicle for a central information-management 
body to “push” relevant content to distributed 
users in real time. 

Cultural Change — Make collaboration and 
knowledge sharing a component of annual 
performance counseling and provide incentives 
(awards and commendations) for initiatives that 
improve collaboration and knowledge sharing. 

Capture and Reuse Best Practices — Best- 
practice teams in analysis divisions or analysis 
units leverage their subject expertise to discover 
and codify best practices, which are then passed 
on to a corporate knowledge-management group 
that helps propagate the use of best practices 
across the enterprise 

Information Expiration Mechanisms — Intranet 
content is automatically identified by date of 
creation and/or last modification; dated 
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documents are flagged and routed to their 
authors, who review them for relevancy and 
decide to archive, delete, modify or reinsert the 
documents back into the corporate database. 

Content Validation Protocols — A process must 
be put in place to prevent the propagation of 
inconsistent or incorrect information throughout 
the enterprise. 

A Strategy for Improving Operations 
Analysis in DoD 

The strategy recommended by the WG to 
improve the practice of military operations 
analysis in DoD through the application of 
enterprise knowledge management is: 

• The Joint Staff, J-8, and OASD (C3I), 
provide leadership in the development and 
implementation of policies that both enable 
and foster knowledge sharing and then 
follow through with the facilitation of 
change to realize knowledge sharing. 

• OASD (C3I) take the lead to inform DoD 
leaders of the value to be derived from 
knowledge sharing in order to change the 
mindset of the leaders. Collaboration and 
knowledge sharing must be promoted within 
the leadership and the community to change 
the culture from protecting knowledge to 
sharing knowledge. 

• DoD organizations and agencies must 
overcome the barriers to knowledge 
management implementation. 

• OASD (C3I) develop policies and 
procedures that result in continuing 
education and training in knowledge 
management practices and technologies. 

The DoD Sponsors should prototype 
collaboration and knowledge sharing processes 
for specific military operations analysis 
problems that may benefit from these 
approaches. These efforts should be used to 
illustrate, from the bottom up, the value of 


collaboration and knowledge sharing in evolving 
the practice of analysis. 

Description and Discussion of WG 1 Findings 
and Recommendations 

There are many obstacles to collaboration and 
knowledge sharing in the pursuit of military 
operations analysis in DoD. Policies that are 
currently in place make it impossible for us to 
use existing knowledge tools. The lack of a 
single coherent GIG or infrastructure precludes 
both technique and technology solutions that are 
tested and available. The pace of technological 
change is daunting; we need a plan and a 
framework for rational decision making that 
allows change to take advantage of marketplace 
innovation while mitigating the impact of 
change on organizations and their budgets. The 
analytic community must find the social or 
economic model that provides the incentives to 
individuals to share information. At its most 
fundamental level, collaboration and knowledge 
sharing will only be a success if it is obvious to 
the individuals involved that it is in their interest 
to participate. Organization and individual 
resistance to required change has to be 
overcome. There are individual and 
organizational incentives in place now that cause 
individuals to hoard information. Individuals 
use the information they have as a means to 
cement their importance. This is probably the 
single biggest hurdle that must be overcome. 
Change is hard; there must be something in it for 
each individual analyst to embrace the concepts 
for collaboration and knowledge sharing. 

There is a need to standardize the practice of 
Military Operations Analysis, that captures those 
activities and procedures that are common to 
most analyses. Standard analytic practices 
facilitate collaboration and knowledge sharing 
by reusing proven methods and analytic product 
frameworks. Standard practices that may be 
accessed through the intranet or Internet also 
help resolve gaps in experience levels that may 
exist in an organization's analytic capability. 

The need for security and the policies that 
enforce security are important, however, 
information security is probably the second 


biggest (after individual ^esist^lnce) stumbling 
block to improved analytical practice. We must 
make a very careful assessment of both the need 
for highly compartmentalized security practices 
(need-to-know) and the way in which these 
policies are implemented. The SECRET 
Internet Protocol Router Network (SIPRNET) is 
already challenging many of the old shibboleths. 
This is not to say we should be careless, but 
rather we must assess the payoff of greater 
sharing versus the risks associated with sharing. 
The rewards for sharing are lowered cost in 
terms of the time it takes to perform analysis. If 
speed of emalysis is as important as our 
leadership insists it is, then barriers must come 
down. 

WG 1 had thirteen specific recommendations 
arrayed across categories of Strategy, People, 
Process, and Technology. They are: 

Strategy 

• The DoD Sponsor should implement 
knowledge management techniques, 
particularly to establish an intranet portal, in 
support of an important study with suitable 
scope, schedule and content that partners 
DoD components MORS, industry and 
academia. The elements that should be 
considered along with strategy include 
people, process and technology. 

People 

• DoD and MORS lead an assessment of the 
capability to provide a baseline inventory of 
analysts for DoD operations research studies 

• MORS assess its capability to establish a 
Knowledge Management Professional 
Development Program for the DoD Analytic 
Enterprise that would include the 
identification of skills and products, the 
refinement of skill levels, and the training of 
analysts. 

• The DoD Sponsors evaluate successful skill- 
inventory systems (e.g. Northrop-Gmmman 
and Y2K) for implementation within the 
DoD Analytic Enterprise. 


Process 

• J-8 accept leadership with OASD (C3I) of 

an evolving analysis process for the purpose 
of improving analysis in a future analysis 
state. Notionally, the evolving process 
could be closely interfaced with an evolving 
knowledge management process as shown in 
Figure 2. 



Figure 2. Interrelationships Related to 
Collaboration, Creation and Reuse 

• DoD determine the business rules for 
contributing analytic knowledge. For 
example, what are the current rules that 
control the access to past studies and how 
could these rules be changed to improve 
access such as those stored at FFRDCs? A 
special case is the identification of the 
business rules for administering and 
controlling an analyst’s portal. 

• The DoD Sponsors, with the assistance of 
MORS, identify new types of information 
required for future analyses (e.g. foreign 
demographics, information warfare related, 
economic information, national 
infrastructure related, chemical/biological, 
etc.) 

• The DoD Sponsors identify current centers 
of expertise required for improved analyses. 

• The DoD Sponsors take steps to systematize 
the organization of information by using 
principles from traditional library and 
information sciences in the electronic realm. 

• The DoD Sponsors, with the assistance of 
OASD (C3I), develop and use an 
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information architecture that structures the 
design of information organization, labeling, 
navigation and indexing systems to support 
both browsing and searching. 

• The DoD Sponsors identify and use a set of 
metrics for the business case preparation and 
performance evaluation of the Analyst’s 
Portal. 

• The DoD Sponsors create knowledge¬ 
sharing incentives such as: Performance 
Reviews that emphasize collaboration and 
knowledge sharing and recognition for 
individuals show collaborate, share 
knowledge and otherwise support the 
organizations efforts in this area. 

Technology 

• DoD concentrate on applying Commercial- 
Off-The-Shelf (COTS) tools as a key critical 
success factor for building analytical 
Knowledge Management portal. Example 
technologies include: 

■ Thesaural Browsers 

■ Personalized Intranet Portals 

■ Federated Knowledge Management 

■ Information-Expiration Mechanisms 

Working Group 2 — Enabling Collaboration 

Vision Statement: Collaborative analysis is an 

analytic activity that involves the sharing of 

data, tools and expertise across organizational 

boundaries to achieve a common goal. 

WG 2 examined four questions: 

• How is collaboration currently used in the 
DoD analysis community? 

• What formal and informal collaboration 
techniques can be applied to improve DoD 
analysis? 

• What analytic practices (local study, 
centrally managed study, etc.) would make 
use of collaboration support? 

• What recommendations do you have to 
improve the existing analytic practice using 
collaboration? 


The working group participants determined that 
the first and third questions were very similar 
and combined their consideration. 

WG 2 Session Summary 

Session 1 - Current State of Collaborative 
Analysis 

Session 2 - When, Where and What Kind of 
Collaborative Analysis is Desired? 

Session 3 - Collaborative Analysis Policy Issues 

Session 4 - Organizing for Collaborative 
Analysis 

Session 5 - Emerging Techniques that Enable 
Collaborative Analysis 

Session 6 - Locating Analysis and Subject 
Matter Expertise 

Where Does Collaboration Exist in Analysis 
Today? 

There are three principal types of analyses in 
which collaboration is a reality today. 

Analysis in the unified commands is 
collaborative by nature due to the military, 
political, inter-agency and international 
considerations. In this real-time environment, 
the decision-cycle is so compressed that analysis 
must be agile to be useful. If tools, data and 
talent are not available when called, the decision 
will be made without analytical insight. 

Analysts in US European Command (EUCOM) 
have learned, for example, that they are 
frequently asked to provide deployment and 
logistical analysis. Collaborative work is made 
more difficult by geographic separation and 
Operations Security (OPSEC) concerns. 

Within the Beltway, collaborative analysis is 
frequently directed to answer Congressional 
Language or to support the Planning, 
Programming and Budgeting System (PPBS) 
decisions. In this environment, joint and service 
considerations pre-dominate. Unlike the 
environment previously described, these 
activities are important, but not so urgent that 
tools, data and skills cannot be massed and 
employed in a deliberate, formal way. A 
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distinguishing characteristic of this kind of 
collaborative work in a hierarchical oversight 
structure that guides the analysis from cradle to 
grave. 

The third collaborative analysis type is the 
service/agency study performed to meet an 
internal requirement. Intra-service and inter- 
service collaboration is driven by the needs of 
the service customer. Analyses of Alternatives 
(AoAs), for example, for Acquisition Category I 
(ACAT-I) systems frequently require a joint 
context. Consultation and rendered assistance 
across organizational boundaries may be formal 
(by memorandum) or informal (by professional 
courtesy). 

Formal and Informal Techniques that Enable 
Collaboration 

Collaboration can be encouraged and assisted by 
both formal and informal mechanisms. One 
motivation for an organization to collaborate is 
to acquire the use of tools and expertise to meet 
infrequent requirements (e.g., support to Joint 
exercises or contingency plan development). 
EUCOM and the Center for Army Analysis 
(CAA) believe that this arrangement works best 
when working relationships have been 
established in advance and are rehearsed. In this 
way, the supporting organization understands 
the decision making processes of the supported 
organization and the hardware/software/security 
requirements for interoperability. While 
collaborative work can be done from a distance, 
EUCOM prefers on-site support. Co-location 
helps allay fears concerning operations security, 
responsiveness and miscommunication. 

The existence of an “honest broker” for data 
encourages participants to share their data 
holdings. The Joint Data Support (JDS) Office 
has earned the trust of the Services, the Joint 
Staff and other offices in the Secretariat, by the 
way, it has controlled the data entmsted to it 
during Mobility Requirements Study 2005 
(MRS-05). Perhaps more important than its role 
as a “gatekeeper,” is the service JDS provides to 
the community by cross-checking the data it 
receives against other authoritative sources to 
identify data anomalies. These anomalies are 


provided back to the source for resolution, 
improving the veracity and consistency of DoD 
data holdings. 

Informal trust-building activities like MORS and 
positive experiences with collaborative analysis 
help to overcome organizational and individual 
resistance to teaming. 

It is important to remember that collaborative 
analysis is not an appropriate technique for all 
situations. A quote attributed to Napoleon is “I 
would rather fight allies than be one.” Sharing 
common ends does not guarantee agreement on 
the ways and means to be used to get there. 

Group pressure to cooperate can result in 
compromises that preserve the status quo when 
what was really needed was creativity and a bold 
shift. 

LTC Jerry Glasow presented a paper in the WG 
that classifies collaborative methods as being 
either “input-based” or “output-based.” MRS- 
05 typifies the former while the Phased Threat 
Distribution (PTD) Study is an example of the 
later. The “input-based” approach reasons that 
if the participants accredit (1) the data, (2) the 
tools and (3) the analysts, the results will, in 
turn, be accredited. When this approach is 
implemented by a sequence of models (e.g., air 
defense precedes deployment precedes force-on- 
force evaluation) a single point failure can delay 
the entire project. It also tends to produce point 
estimates rather than a range of possible 
outcomes for consideration. 

Two advantages of the “output-based” approach 
are parallel, independent analysis and multiple 
results that can be compared. The most difficult 
part of this approach can be determining why the 
results are different. If this approach is used, 
several integrated teams should be 
commissioned by the participants rather than 
putting the participants in competition with one 
another. 
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Data Collection and Production 

Data collection, production, preservation and 
maintenance were recurring themes in the WG 
deliberations. Jim Methered, citing his 
experience during and after OPERATION 
ALLIED FORCE, is vehement that data 
collection must be a component of the Joint 
Task Force (JTF) information management plan 
instead of the “ad hocery” that exists today: 
“...scavengers picking over small, rapidly 
decaying, incomplete and questionable data and 
then fiercely defending the scraps gathered from 
others.” 

A potential major future role for operations 
analysis may be to study how infonnation is 
used and develop ways of managing and using it 
better. Process engineering in peace might 
facilitate analysis in war as well as improve 
actual and simulated decision making processes. 

Description and Discussion of WG 2 Findings 
and Recommendations 

The working group recommendations are 
collected under three headings, people, process 
and products. 

People 

We must find ways to identify (and evaluate) 
subject matter experts to make access to such 
people fast and to provide confidence to the 
analyst that the SME they are in contact with is 
respected in the subject matter domain with 
which they are associated. 

We must find a means to develop operational 
experience in Civil Service 1515s and uniformed 
analysts who do not hold warfighting specialties. 
The alternative is to find a way to integrate non¬ 
analyst operators in analysis efforts. 

Process 

There is a spectrum of collaborative analysis 
processes, and each has its strengths and 
weaknesses (bold shifts versus consensus). 

• We can organize to perform competitive 
analyses to find new ideas. 


• We can organize to search for a “common 
ground” solution. Collaborative analysis is 
a “team sport” and requires practice. 

Operational Analysis support to Commanders in 
the field is most effective when a mindset of 
“Train as you Fight” is embedded in training and 
analysis practice. Ways must be found to re¬ 
establish formal liaison/exchanges to improve 
the quality of analysis. 

Today, collaborative analysis takes longer to 
perform than analysis done in isolation. In the 
future, timelines can be shortened if 
informational exchanges and liaison between 
analytic organizations/agencies are 
institutionalized in our business practice. 

In the short-term, it’s important that we 
reconcile our demands for data with our ability 
to produce it. Dr Chris Lamb, ASD(S&TR), is 
currently leading an effort with the services and 
the intelligence agencies to better coordinate our 
efforts. It also makes good sense to plan in 
advance for the long-lead data that are common 
to many of the studies we do (e.g.. Time-phased 
Force Deployment Data (TPFDDs)). In the 
long-term, data collection in the operational 
environment needs to evolve from something 
that interferes with task performance to 
something that is transparent to people 
performing their day-to-day tasks. Similarly, 
directives to perform large, collaborative 
analysis needs to be harmonized with the day-to- 
day analytic workload. 

• Support independent, accredited sources for 
data to speed collaborative effort. 

• Find ways to identify data to source new 
applications and phenomena. 

• Data collection must be deliberate and done 
in such a way that it does not require 
additional effort by operators. 

• Synchronize data production with data 
demands and institutionalize production and 
update of long-lead items (e.g., extend the 
Joint Munitions Effectiveness Manual 
(JMEM) to include developmental systems). 
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• Harmonize schedule of major collaborative 
projects and service workload. 

Products 

Compressed decision-cycles and the tendency 
toward using briefings rather than reports as 
deliverables has lessened our use of literature 
search and our discipline to thoroughly 
document what we do. 

We need improvements in literature search 
capabilities, identifying work in progress that 
may affect the project at hand. 

We must improve our study documentation 
techniques, allowing for reproducibility of 
results and reuse of data in addition to 
descriptions of our procedures and findings. 

Working Group 3 — New Analysis Challenges 

WG 3 examined a single focus question: 

• What new military operations problems 
should the analysis community consider in 
its evolution of analytic practice? Or stated 
differently, what problems are not well 
served by the current state of the practice? 

The working group examined military 
operations analysis questions for which the 
established military operations analysis 
community apparently does not have the 
answers. The working group was about new 
problems, problems not involving conventional 
warfare, and/or post-Cold War problems. 
Actually there are old problems that belong here 
as well. The issue isn’t whether a problem is 
new or old, but rather whether the military 
operations research community has succeeded in 
addressing it satisfactorily. An example is the 
problem of determining the value of a common 
operational picture to the warfighter. 
Presentations to this working group came from 
the major analysis players such as the Services’ 
Studies and Analysis shops, from warfighters 
like the US Joint Forces Command (JFCOM) 
and US Pacific Command (PACOM), and from 
decision makers like OSD, the Joint Staff and 
the Services' warfare requirements and 
assessment organizations. 


WG 3 Session Summary 

Session 1 - New Analysis Challenges 

Session 2 - Challenges in Army Modeling 

Session 3 - Analysis Challenges for Joint 
Operations - A USCINCPAC Perspective 

Session 4 - New Analysis Challenges in Navy 
Programs and Fleet Battle Experiments 

Session 5 - New Analysis Challenges in Air 
Force Programs and Logistics Analysis 
Challenges 

Session 6 - Analytical Challenges of the USMC 

Analysis Problems that are not Well-Served 
by Current Analysis Techniques or 
Approaches 

With the end of the Cold War, the Department 
of Defense and thus the DoD analytical 
community, are faced with the demand of 
conducting and thus analyzing a variety of new 
operations. These new operations require new 
methods of analysis for planning, conceiving 
concepts of operations, defining doctrine, and 
analyzing effectiveness. Some examples of 
these new domains are: 

• High Intensity/Low Casualty Conflict. 

• Military Operations Other Than War 
(MOOTW). 

• Humanitarian Operations. 

• Peacetime Engagement. 

• Operations in Urban Terrain. 

• Technology Insertion (Digital Battlefield, 
Tomorrow’s Dismounted Warrior). 

• Chemical/Biological Warfare (Including 
Contagion). 

• Asymmetric Threats (domestic terrorism, 
political manipulation, homeland defense). 

• Special Operations. 

• Information Operations/Warfare. 
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Problems with the Current Emphasis 

The current emphasis on modeling and 
simulation with its resulting de-emphasis on 
other analytical tools and methods is limiting the 
analysis toolkit. With the amount of time and 
resources required for scenario development and 
data collection to run the current models and 
simulations, some other alternatives are clearly 
needed. For example, methods of scoping 
analysis that can quickly look at the alternatives 
and potentially set the stage for further, larger, 
study are needed. These scoping tools can serve 
to pare down the alternatives and allow the focus 
on the important question to evolve. In addition, 
simpler analysis tools such as spreadsheets 
should be used and exchanged within the 
analytical community as a rapid means of 
knowledge transfer. Finally the analyst should 
be taught to combine the results from all 
pertinent tools and analytical methods to more 
thoroughly, accurately and quickly answer the 
question at hand. 

With the increasing complexity of today’s 
military scenarios and high technology weapons, 
there is a greater need for the analysis 
community to be able to identify and 
communicate the effects of the factors and their 
interrelationships. It is especially important to 
be able to conduct and communicate the trades 
conducted between factors that typically would 
not be linked. For example, an aircraft carrier 
can be the platform used for Joint Strike Fighter 
(JSF) launches, and a typical analysis may look 
at the number of targets put at risk by various 
sizes of JSF fleets. However, an interrelated 
measure that should be considered is the deck 
space that is being taken up by the JSF fleet, that 
may potentially be used for such things as 
helicopter medical evacuation (MEDEVAC) 
support, two very important missions with 
interrelations that must be considered. 

There are many factors that effect the outcome 
of a battle, but do not have directly measurable 
combat effects. These factors must be taken into 
account, and their effects must be traced to 
operational and strategic outcomes. These 


factors include such things as: 

• Logistics. 

• Bandwidth requirements. 

• Engagement/presence on PERSTEMPO and 
OPSTEMPO. 

• Infrastructure investments. 

• Readiness postures. 

• Various force structure alternatives. 

The behavior of individuals, small units and 
adversary leaders must be captured in our 
evolving analytical tools. The effect and 
measurement of behavior factors at the 
individual or small unit levels are important 
considerations, especially in the analysis and 
evaluation of combat systems. For example, the 
increase in situational awareness for the 
individual foot soldier is probably a good thing 
to a point, until information overload occurs or if 
the equipment used to give the situational 
awareness decreases the individual’s 
effectiveness in other areas, such as mobility. 

On the other hand, being able to quantify the 
characteristics and motivations of an adversary 
leader would allow the identification of the 
leader’s or country’s centers of gravity and may 
assist the planning process in identifying 
effective target sets that would influence the 
enemy’s course of action. 

Methods for quantifying risks of all sorts must 
be developed and institutionalized. 

The frequency of calls for quick-tum around 
analysis is increasing. These analyses take 
many forms, such as support for testimony to 
Congress or support to ongoing warfighting 
efforts. To answer these types of questions, the 
analyst needs a set of tools that support these 
quick analysis activities. These tools need to 
capture the important elements that would be 
drawn out by more thorough analysis but run 
quickly enough to support answering the 
question when the answer will still be desired 
and relevant. 
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Description and Discussion of WG 3 Findings 

and Recommendations 

There were three broad recommendations that 

were defined by WG 3. 

• As the analytical community evolves, it 
needs to find tools to deal with new forms of 
operations, find methods to trace known 
factors and interactions to their logical 
combat output, and find tools that will 
provide accurate and thorough analysis in a 
short timeline. 

• The DoD analytical community needs to 
nurture analysts and emphasize analytical 
thinking rather than focusing on simulation 
and modeling development. The community 
must project a more balanced view of model 
capabilities as well as limitations, and 
reverse the recent trend toward blind faith 
toward large models as the solution to all 
analytical problems. 

• The analytical community needs to find 
means for characterizing and 
communicating the impacts of uncertainty. 
For example risk, such as decision risk 
based on the fact that various factors may 
not have been considered in the current 
analysis, need to be characterized and 
communicated to the decision maker. 

Working Group 4 — New Analysis Tools and 

Methods 

WG 4 examined four questions: 

• What currently emerging developments in 
analytical approaches to complex systems 
can be applied to challenging military 
operations analysis problems? Consider 
areas such as chaos, complexity theory, etc. 

• To what DoD analyses, particularly the new 
analytic challenges defined by WG 3, can 
these approaches be applied? 

• What recommendations do you have to 
improve the existing analytic practice in the 
near term by using new analysis tools and 
methods? 


• What recommendations do you have for 
areas of research for the next generation of 
military operations analysis? 

The New Analysis Tools Working Group sought 
to identify emerging tools and techniques for 
conducting operations analysis, given the new 
challenges that face the military operations 
research community in the 21st century. 

To accomplish this, MORS leadership 
assembled representatives from domains that are 
not normally part of the traditional MORS 
population. Experts with expertise in 
Command, Control, Communications, 

Computers and Intelligence (C4I) systems, 
intelligence, policy analysis, artificial 
intelligence and knowledge management were 
present. This eclectic set of skills resulted in 
information exchange that exceeded 
expectations, a vision focused on providing a 
foundation of consistent and compatible data, as 
well as an architecture that could provide 
connectivity between command and control 
systems and analytic modeling environments. 

The kinds of connectivity and consistency that 
are envisioned in this product are believed to be 
absolutely essential for providing commanders 
and staffs in the field with flexible automated 
planning support, and keeping analysis relevant 
in the highly uncertain future that the DoD faces. 

Perhaps most importantly, it was concluded that 
for the most part there is no technological 
impediment to achieving the vision that is 
articulated herein, and that these kinds of efforts 
are an essential part of achieving the knowledge 
management “culture” that has been 
implemented successfully in many commercial 
enterprises. 

WG 4 Session Summary 

Session 1 - Mapping “New Tools” to “New 
Needs” 

Session 2 - Joint Warfare System, Joint 
Modeling and Simulation System, Land 
Formation Model 
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Session 3 - Army Modeling and Simulation 
Office, Army Artificial Intelligence Center, Joint 
Data Support 

Session 4 - C4I Model, Project Albert and 
Operational Synthesis and DTI 

Session 5 - Mapit and Dynarank 

Session 6 - IWEDA and Terrain Modeling 

Assumptions and Scope 

Because the solution space within which the 
working group was operating the group made 
initial assumptions to bound its discussions. In 
some cases the assumptions were found to be 
too restrictive and were later relaxed. 

The first assumption is that Knowledge 
Management experts will create a repository 
infrastructure, and that infrastmcture will be 
available for use to store and distribute analysis 
domain knowledge. 

Analysts within DoD have been modeling and 
simulating for a long time. There is an existing 
set of recommendations in the DoD Modeling 
and Simulation Master Plan. These 
recommendations are assumed to be good, and 
future recommendations should deviate only 
when needed. 

There are architectures available to us to move 
information that would be in a knowledge 
management oriented system and culture. 
Discussion with experts from the C4I domain 
revealed that the assumption does not hold with 
respect to the Defense Information Infrastmcture 
Common Operating Environment (DII COE) 
standards. 

The Analysis Functional Working Group of the 
DoD Executive Committee for Modeling and 
Simulation (EXCMS) has a valid list of analysis 
deficiencies that needs to also be considered. 

The WG focused on how to do big studies better 
and then began to look at minute-long decision 
cycles (from WG 3) to balance out the 
discussions. 


Analytical Requirements 

Although warfare doctrine and military missions 
are evolving, there are constant / recurring 
analytic requirements that can and should be 
planned for: 

• Force assessment. 

• Planning and execution. 

• Deliberate planning. 

• Crisis action planning. 

• System effectiveness and trade off analysis. 

• Concept and doctrine development and 
assessment. 

These missions are fairly constant in the 
workspace of military operations research. There 
is ever reason to believe they will continue to be 
the main missions of the future. They form the 
space around which to stmeture the new tools 
for analysis challenges. 

Because the group began deliberation at the 
same time WG 3 was meeting, a generic set of 
decisions that could be supported by analysis 
was used to stimulate discussion. Over the 
course of the workshop, the group considered 
WG 3 inputs as they arrived. 

One notable possibility that was added by the 
C4I domain was the existence of decision aids 
that are either currently embedded, or planned 
for in near real-time command and control 
systems. These systems would benefit from 
analytically sound underlying algorithms and the 
group felt strongly that the analysis community 
would need to be an active part of their 
development. 

Accomplishing these analytic missions requires 
descriptive tools (models and simulations), 
prescriptive tools (choice and resource 
allocation), reliable and consistent data. 

A significant body of work is underway in the 
DoD that advances the state-of-the-art in 
modeling “infrastructure.” This work seeks to 
correct deficiencies inside mission areas. 
However, it fails to capitalize on related work 
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(schedule and resource constraints). This results 
in inefficient and inconsistent modeling and 
analysis efforts. 


A Knowledge Management Framework for 
Analysis 


During the conference, we saw the knowledge 
management framework (the dark gray or blue 
boxes) as a hierarchy of knowledge products 
built on the base of known facts and derived 
meanings as shown in Figure 3. 
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Figure 3. Analysis Process in the Context of 
Knowledge Management 

One of the insights from our work is that the 
application of knowledge management within 
the analysis community is actually a process of 
becoming conscious of the importance of 
building and sharing the knowledge of our 
analysis enterprise — both subject area and 
analysis expertise. 

The analysis process (shown in light gray or 
yellow) produces knowledge products. The 
essence of analysis has always been to create 
knowledge from the raw materials of facts and 
meanings. 

One aspect of wisdom — the highest level 
knowledge product shown in the original 
framework (and missing from the far right side 
of this diagram) — results from continuing to 
develop and use analysis capabilities over time. 


Figure 4 provides a slightly different view of the 
analysis process and products from the previous 
slide. It highlights the product areas that the 
working group might consider as areas for 
improvements. 
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Figure 4. Analysis Process Alternative View 

Much of the discussion centered on tools and 
data, particularly the need to integrate 
information generated from multiple tools, 
sequence data flows from one tool to another, 
and to ensure that the data objects were 
consistent such that they could be used as is or 
logically transformed to meet modeling 
requirements. The term “model,” defined in the 
broadest sense as the abstract representation of 
any problem, was frequently discussed as a key 
tool. 

Another emphasis area was to reduce the data- 
to-tool-to-information cycle time sufficiently to 
meet the needs of the customers with very short 
decision timelines. Traditionally, these 
customers have relied on back-of-the-envelope 
estimates, “quick and dirty” analyses, to offer 
insights to support decisions. Since these 
decisions are frequently high-level and wide 
ranging, the analysis community would like to 
design tool and data solutions that offer both a 
high quality and timely response. 

Decision support needs centered on visualization 
tools — the ability to clearly represent complex 
decision spaces, dissimilar tradeoffs and the like 
— that have not yet been identified. The group 
identified knowledge management practice as 
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the preferred solution option to identify and 
reuse methodologies and measures. 

An Architecture of Systems, Software and 
Models to Support Analysis 

The group developed an “architecture” of 
systems, software and models that was perceived 
to be necessary to meet the needs of military 
leaders, and defense managers in the twenty-first 
century. This architecture is shown in Figure 5. 
The kinds of unprecedented connectivity and 
consistency that are envisioned are required to 
keep analysis and analysis products relevant 
inside the short decision cycles that are 
emerging. It is also necessary to achieve a 
culture of “Knowledge Management” in which 
the system of systems we are attempting to 
analyze has achieved a complexity that is 
beyond the comprehension of any one person or 
discipline. 



Figure 5. The Future “Framework” for Analysis 

There are three primary nodes in this network of 
systems. First, there are near real-time C4ISR 
systems. Operational planners and commanders 
who are tasked to execute the nation’s military 
missions use these systems. In the “information 
age” military, they exist in some form or another 
from the National Military Command Center 
(NMCC), down to the soldier in the foxhole. 

The second node is analysis models and 
simulations. These models and simulations, 
ranging from simple spreadsheets and decision 
tables to large theater simulations are used to 
develop and evaluate plans, choose courses of 


action, and manage the DoD. The final node 
envisioned consists of training simulations. 
Again, these simulations vary in scope and 
complexity. 

All of these nodes have a common requirement. 
They all need data (both primitive and meta) to 
function. If we are to achieve the envisioned 
interoperability it is absolutely essential that this 
data be logically consistent, and portable across 
domains when desired. 

One fact that emerged is that the C4I community 
is embedding decision aides into emerging 
command and control systems. These decision 
aids are effectively models that operational staff 
members and commanders will use to make 
decisions on those short timelines we see 
emerging. It is important that analysts (and 
others) be involved in the development of these 
algorithms to increase the likelihood that the 
underlying logic is sound and consistent. 
Moreover, this kind of involvement is necessary 
if the DoD is to achieve its objectives with 
respect to the reuse of software, and achieve a 
more cost effective maintenance stream for 
models and data of all types. 

In theory the DII COE standards provide vehiele 
for data communication from live systems to 
simulations. However, it was learned that the 
communication between the C4I and analytic 
communities will not support this objective. 
Moreover, there are inconsistencies between the 
C4I and training domains. 

An example illuminates how this system would 
improve our operational decision making. 
Envision a requirement for PACOM to deploy 
humanitarian assistance to East Timor. There is 
a need to provide logistical infrastructure to 
rebuild or secure this nation. Also, as.sume the 
planning process produced a contingency plan 
that includes identification of the forces that 
would be sent on this mission. 

The plan and all of its assumptions have been 
described in some analysis model (perhaps in an 
Operations Plan (OPLAN) Format) and reside in 
data. At the time the requirement to execute is 
identified, the data in the plan are compared to 
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ground truth data. The army database on current 
forces identifies the fact that the 25th Supply 
and Transportation Battalion which was in the 
plan is not available because it is deployed. 

And, the US Transportation Command 
(TRANSCOM) data shows that the C-5s that are 
necessary to meet the timeline are committed to 
some other high priority mission. What has 
happened is that we have used existing data to 
quickly illuminate assumptions that were valid 
at planning time, but are not at execution time. 
Operators are then quickly able to focus on 
managing the exceptions. 

The deliberations in the group revealed that 
there are fundamentally no significant 
technological impediments to achieving this 
vision. The obstacles are largely related to 
achieving a sound common data model, and 
developing a cultural will to expose data to other 
domains. 

Description and Discussion of WG 4 Findings 
and Recommendations 

The working group had five broad 
recommendations associated with the analysis of 
new tools and methods for analysis. 

• Increase DoD emphasis on supporting inter¬ 
service collaboration toward standards. This 
builds on the existing service efforts. 
Existing organizational models include the 
Joint Technical Coordination Group’s 
(JTCG) (JMEM) and the Modeling and 
Simulation Resource Repository (MSRR) 
Board of Directors. This should include 
Joint mles for reconciliation of standards. A 
baseline should be established (i.e. a 
minimal standard for meta-data). 

• Use the accumulated but limited experience 
as a foundation for analysis knowledge 
management implementation. Establish a 
formal plan for achieving the vision 
including senior leader support and 
involvement, a deliberate investment 
strategy, and one that uses the multi¬ 
disciplinary team. There should be a co¬ 
evolution of culture and funding for 
technology. 


• The analysis community should be 
represented in the DII COE process. The 
on COE standards should be impacted by 
requirements that emanate from the analysis 
community. 

• DoD should develop and resource a strategy 
for collecting “other source” data, and 
applying knowledge management. Other 
source data include National Training 
Center (NTC) exercise data, historical 
deployment data and lessons learned data. 

• Encourage the creation and use of “quick 
and dirty” tools. This should be 
accompanied by: the development of mles 
of the road for using data, the accumulation 
of best practices and mechanisms for timely 
peer review, the use documentation and 
allowance for evolution. 

The Synthesis Group Report 

Vision Statement: Since analysis is both a 
product and a process, our vision is a re¬ 
emphasized and revitalized practice of analysis 
to more effectively and more efficiently connect 
the decision maker and analyst so that the final 
product is not the primary result. 

The Synthesis Group included members from 
each working group and met daily with the 
chairs and co-chairs of those groups. This was 
done to both highlight common areas and to 
request effort on specific topics that would help 
the Synthesis team in their efforts to create a 
follow-on action plan to help the military 
operations research community achieve the 
overall vision of the conference. 

Toward this end, the Synthesis Group asked 
each WG to articulate a vision that would 
capture the end state resulting from full 
implementation of their group’s 
recommendations. Based on their inputs, the 
Synthesis Group articulated its own Vision 
Statement. The Synthesis Group believes this 
vision transcends and incorporates the subject 
area working groups’ visions and describes how 
emerging capabilities such as collaboration and 
knowledge management can transform the 
practice of military analysis. The emergence of 
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powerful communication and computer-based 
capabilities is already changing the economic 
world. Where before the focus was on 
production, the emerging giants of e-commerce 
are now succeeding through their acquisition 
and management of information. The most 
successful and most profitable of these 
businesses are those that enable the highest level 
of information sharing, so that opportunities are 
quickly identified, decisions made based on the 
full amount of information and insight available 
in the work force, and then action initiated 
appropriately. The knowledge resident in every 
worker is rapidly connected to the decision 
makers so that timelines are reduced and 
customer needs satisfied, optimizing company 
effort and taking full advantage of the 
knowledge base of the organization. In a like 
manner, military analysis in the future will need 
to be fully integrated, with analysts and decision 
makers so closely linked through technical and 
personnel means that the maximum benefit will 
flow to the organization. In fact, the interaction 
will be such that the final results of any study 
will surprise no one, since decision makers and 
analysts will both be engaged throughout the 
analytic process, allowing the strengths and 
insights of each to be fully incorporated. Such a 
construct will revitalize the practice of analysis, 
take advantage of the emerging capabilities, and 
better address new challenges confronting 
analysis in the new millennium. 

The Tool and Challenge Matrix 

The products of WG 3 and 4 analysis were 
combined in Figure 6 to illustrate what new or 
emerging tools and techniques show promise to 
address, at least in part, the challenges identified 
by WG 3. 

Those challenges fell into six broad categories: 

• Changing characteristics of warfare in the 
21st Century (New Wars). 

• Inclusion of the effects of humans and 
human factors (e.g., moral, training, etc.) 
into models (Behavioral). 


• Showing the effect of logistics and other 
support activities on warfighting outcomes 
(support). 

• Identification of complex interrelationships 
between input variables and between 
internal variables (processes) within models 
(interactions). 

• Explicit quantification of risk and 
uncertainty in analysis and in 
communications with decision makers 
(risk). 

• The challenge of developing and gaining 
acceptance of analysis techniques that are 
not dependent upon traditional simulation or 
other models (e.g., linear programs) (non 
model). 
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Figure 6. Tool / Challenge Matrix 


WG 4 identified many recommendations 
regarding new or emerging tools and techniques 
that should be explored and implemented where 
appropriate. These tools and techniques fell into 
ten categories as shown in Figure 6. 

The first category listed involves standards. 
There are some relatively new efforts underway 
to develop common standards that can be 
applied to the models and data used to support 
analysis; the Army Standards Category 
Coordination process is one example of this. 
Defense Modeling and Simulation Office 
(DMSO) -sponsored efforts to develop common 
conceptual models and the MSRR are other 
examples. The group recommended that the 
fundamental algorithms underpinning many 
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major simulations in use be examined in open 
(peer review) forums and reconciled. Although 
a commonly accepted set of algorithms would be 
desirable, it is not necessary that all services or 
other users agree to a single representation of a 
warfare phenomenon. We can agree to disagree, 
provided the differences are clearly documented 
and understood and made available to the 
analysis community. 

The second technique identified by WG 4 is 
automated data generation. The biggest single 
complaint from the WG was the lack of needed 
data and the difficulty and cost of generating or 
obtaining data even when it is available. 
Automated means of generating commonly used 
data would increase the responsiveness and 
reduce the cost of getting the data. 

The “Analyst’s Rucksack” is an example of 
using quick and simple tools and decision aids to 
fill in for large model deficiencies and directly 
support warfighters. The Workshop endorsed 
the development of tool sets of this nature. 

The commercial world is also producing an 
array of tools that have great utility for 
operations research analysts. Tools which have 
been around for several years, but are becoming 
increasingly sophisticated and easy to use, 
include statistical and data analysis packages, 
linear and nonlinear programming, process 
simulation and decision analysis software. 
Emerging products include neural networks, 
Bayes nets, colored petri nets and other software 
that propagate inferences or influences. 

There is recent research that is producing some 
new techniques that may be applied to fix, in 
part, some of the areas that lack tools for 
analysis. Agent-based simulation may help in 
the analysis of doctrine, procedures and risk 
sensitivity, and recent mathematical models 
have been developed to assist in the simulation 
(and real-world execution) of fusion between 
multiple sensors and sensor disciplines. This is 
not an exhaustive list, but these are two areas 
that show promise. 

Performing analysis in the 21st century will 
likely lead to the resurgence of the multi¬ 


disciplinary team. However, this team will have 
a different composition that the teams in the 
early days of military operations research, which 
were largely formed from hard sciences. 
Although some problems may still require 
physicists, chemists, etc., most work will require 
groups pulled from three broad areas. The first 
consists of domain experts, which will often 
include warfighters, doctrinal experts and other 
expertise in military operations. The second 
group is software and computing experts, as 
models and tools become increasingly complex 
from a software perspective. The third group, 
which helps to meld the other two together, 
consists of operations research analysts and 
other experts from related analysis domains, 
such as mathematics and statistics. We should 
no longer expect that a single military operations 
research analyst be able to read and write code, 
develop mathematical models and possess a 
broad military background to include joint 
operations. 

The implementation of knowledge management 
techniques identified by WG 1 and presented by 
plenary session and lunchtime speakers will also 
help address the challenges to the practice of 
military operations research. These concepts are 
developed in detail in other sections of this 
report. 

Modeling and Simulation Executive Agents 
(MSEA) have, in some areas, greatly advanced 
the state of the art in modeling and simulation 
(M&S). The environmental MSEAs have had a 
great degree of success and can serve as 
exemplars of the process that leads to significant 
improvements. They have developed data 
standards applicable to a wide variety of 
simulations (to include both training and 
analysis simulations) and real world command 
and control systems, and populated these data 
structures with data that is easily obtained 
through user-friendly business processes. They 
have also developed algorithms to transform the 
data into information useful to aggregated 
models; for example, the Army has developed 
techniques for automatically generating road 
networks from detailed NATO-standard 
mobility networks. 
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One area that can be addressed through good 
knowledge management practices involves 
“other source” information. This data includes 
NTC exercise data, historical deployment data 
and “lessons learned” data. These data 
sometimes exist but are not necessarily recorded 
for posterity (e.g., much NTC information is 
outbriefed to the unit commander and then 
deliberately destroyed), not captured at all (e.g., 
historical deployment data), or are captured only 
qualitatively rather than quantitatively (e.g., 
lessons learned). These data, if resources are 
applied for capture and storage, can significantly 
improve our ability to validate models against 
“real world” experiences.^ 

Finally, the group recommended that additional 
research be made in the area of automated 
decision making. Large simulation models 
require some automated representation of a 
human decision maker (e.g., a commander and 
staff in an analysis model or an opposition force 
(OPFOR) or subordinate commander in a 
training model) to provide reproducibility or 
reducing exercise/training costs. However, the 
current implementations are recognized to be 
insufficient and unrealistic. Automated decision 
making can also be embedded in real-time 
decision aids, not to replace a human decision 
maker but to advise him or her. 

Analysis to Support Decision Making 

Figure 7 illustrates one view of the current 
practice of analysis. 


It is recognized that exercise data is not real combat data, 
but it is useful surrogate data that may be more accurate 
than outputs from abstract models and simulations. 



Figure 7. Analysis to Support Decision Making 


It shows that analysis is focused on one goal: 
helping decision makers choose among 
alternatives. The four activities analysts support 
in this practice include: 

• Helping the decision maker define the issue 
or questions. 

• Helping to define the decision space so that 
a full range of alternatives can be 
considered. 

• Establishing meaningful measures of 
effectiveness and criteria that will help show 
the strengths and weaknesses of the various 
alternatives. 

• Then creating understandable, traceable and 
repeatable methodologies that will assist in 
this selection process. 

To do this, an analytic foundation must be in 
place. This foundation includes: 

• People who contribute to the process, 
primarily analysts, but including a broad 
range of other scientists, engineers and 
SMEs. 

• Models, or the concepts, mathematical 
abstractions or other constructs that are used 
to decompose a problem and focus on key 
aspects. 

• Data, scenarios, assumptions and other 
factors that are necessary to support the 
process. 
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The issue addressed by this workshop is how 
this framework is, and will be, impacted by the 
changes in the national security and 
technological landscapes and how the practice of 
analysis should evolve to handle the challenges 
and take advantage of the opportunities provided 
by these changes. 

Of course, the very complexities presented by 
the rapidly changing landscapes associated with 
this environment may have removed the ability 
for analysts and others to focus on the big 
picture. The new environment as assessed by 
WG 3 presents a conundrum; we are asked to 
provide insight on the complex interrelationships 
between factors input to and within our models 
(that reflect interrelationships in the real world), 
but we are also asked to reduce the complexity 
and increase the transparency of our tools and 
analysis. Although it may not be possible or 
necessary to reduce the complexity of our high- 
end models (recognizing of course that not all 
analysis requires large and complex models), 
open sharing of model data, assumptions, 
algorithms, etc., all within a set of agreed-upon 
standards and implemented through knowledge 
management, may increase the transparency of 
the models and analysis process among trained 
analysts. The analyst will always have to 
convert complex mathematical and structured 
logic into forms understandable to a decision 
maker without an operations research 
background in all but the simplest cases. Thus, 
an elegant solution may not be a reduction in 
complexity to a handful of variables, but instead 
be a transparency and sharing among analysts 
that is communicated to the decision maker 
throughout the analysis (rather than at the end) 
in a clean and direct (if not elegant) manner. 

At the same time, there are potential solutions 
available, at least to address part of the changing 
environment, and to help our analytic 
community and other parts of the DoD 
enterprise to adapt and prevail in the emerging 
environment of the 21st century. Dr Ruth 
David, the first keynote speaker for this 
workshop, defined Knowledge Management as: 

• The process through which an enterprise 
uses its collective intelligence to achieve 


strategic objectives. Furthermore, 

Knowledge Management is realized through 
strategy, people, process and technology. 

Inherent in her definition was the understanding 
that the enterprise using knowledge management 
was a “lean, mean, fighting machine” team 
focused on common objectives and with a 
shared understanding of the necessity and value 
of each other’s contribution. Such a definition 
of enterprise may not reflect the current state of 
DoD, where mutual contributions are not 
recognized and parochialism can break down 
teamwork. However, to the extent that shared 
goals and teamwork reflect DoD, and in the 
areas that knowledge management can help the 
department move toward this enterprise vision, 
the department should surely embrace 
knowledge management and its implementation. 

Certain aspects of knowledge management can 
already be endorsed in ways that do not threaten 
existing culture. In particular, knowledge 
management has the ability to help us 
distinguish between what we know and what we 
don’t know by helping the enterprise quickly tap 
the collective knowledge of the individuals in 
the organization. Done correctly, such a 
capability will allow an individual assigned to 
work a problem the capacity to poll all experts in 
or out of the organization to assess what might 
already be known or what is in the process of 
being studied about the problem. Rather than 
wasting time reinventing wheels, a properly 
implemented knowledge management system 
will save enormous time and resources by 
helping the analyst and others rapidly determine 
what is already known as the basic starting point 
of any study. The same system, by enabling 
rapid connectivity between analytic practitioners 
will also enhance information sharing and 
collaboration. When one study team, already 
focused on a specific area, realizes that the value 
of its research can be increased by adding just a 
small amount of effort in order to address a 
broader range of questions and meet the needs of 
a broader customer base, one promise of 
knowledge management will be realized. In 
fact, as the limited resources available in an 
organization self-organize to focus on the areas 
they do best in order to provide a broader base 
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of insight and information for the organization, 
the people involved will clearly understand that 
their value-added is not in developing and 
hoarding information, but rather in sharing 
information in ways that enhance the overall 
capabilities of the enterprise. Value-added 
comes not from having information since in 
today’s world information will be available to 
everyone. The value instead comes from the 
ability to use available information in new and 
creative ways to enhance the capabilities of the 
organization. 

Returning to our analytic construct, we can see 
that changes in the national security landscape 
are already shaking some of the analytic pillars. 
Evolving from attrition-based warfare to effects- 
based warfare, understanding the implications of 
Command, Control, Communications, 
Computers, Intelligence, Surveillance and 
Reconnaissance (C4ISR) systems and 
capabilities, and the emergence of a broad new 
array of asymmetric threats are all impacting the 
ability of the analytic community to develop 
reasonable measures of effectiveness as well as 
to develop clear and repeatable methodologies to 
assess these aspects of military activity. 


Figure 9 provides more detail on the changes, 
with the circled “national security landscape” 
reflecting the circled area on Figure 8. We’ve 
already hinted at the changing paradigm of 
massed armies meeting on a battlefield to the 
massing of effects at the appropriate time and 
place as the way current US military leadership 
is starting to think about warfare. In fact, to help 
think through the implications of such changes, 
MORS has conducted a number of workshops 
and will hold a C4ISR Follow-on Workshop in 
the fall to continue to evolve our understanding 
in this arena. 


•'•.National Security Landscape.*,** 

- Massing forces -> massing effects (C4ISR follow-on 
meeting) 

- Environmental - Threat (World is changing - are we?) 

- Ash Carter & Bill Perry (A/B/C Lists from the QDR 
Workshop) 

- Technology Insertion 

- QDR 2001 (strategy - ???) 

• Technology Landscape 


- Knowledge Management 

- Collaborative Analysis Management 

- Commercial Initiatives 


While these areas are outlined with the dashed 
lines (which appear in red in color versions of 
Figure 8), the technological landscape and 
emergence of new capabilities like Imowledge 
management are impacting the rest of the words 
to potentially transform the way we conduct 
analysis. 



Figure 9. Challenges 

Some of our speakers in this workshop have 
referred to the Ash Carter/Bill Perry work on 
who the threats of the future are. They range 
from an A List of direct threats to national 
survival which would include homeland attack 
by catastrophic terror agents, to a B List of 
threats to national interests that the current 
strategy seems to address, to a C List of 
important problems that do not threaten US vital 
interests (e.g. Kosovo, Somalia). In addition, 
new technologies require new analytic 
capabilities to measure the effects and potential 
contributions from such areas as directed energy, 
information technologies, unmanned systems 
and space. The QDR process may change the 
national military strategy, so the analytic 
profession will need to be responsive both in 
helping to determine the implications of strategy 
shifts as well as in implementing any new 
strategy in the analyses of the future. 
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Just as the national security landscape is 
changing, so is the technology landscape in 
which analysis is conducted. This workshop has 
highlighted knowledge management and 
collaborative capabilities, but there are a number 
of other initiatives, such as simulation-based 
acquisition and other e-commerce activities that 
the analytic community needs to track and adopt 
where it makes sense to do so. 



Figure 10. Potential Future “Framework” 

Figure 10 shows an illustration of an ideal end- 
state for the current efforts to improve analytical 
practices. There are three families of models and 
simulations: M&S embedded in real-time 
corrunand and control systems; training and 
exercise support simulations; and models and 
simulations used to support analysis (to include 
Test and Evaluation (T&E) analysis and analysis 
of non-warfighting issues such as infrastracture 
and business process improvement). Within each 
domain are a set of models and simulations 
tailored to the purpose at hand. No single model 
will apply universally across all domains, but in 
this ideal end state we have data, software 
objects and process representations shared 
across the domains where it is logically 
consistent with the model and purpose. 

Infrastructure programs currently under 
development are assumed to be in place in this 
end state, to include the High-Level Architecture 
(HLA) for simulations and the DII COE linking 
all areas together. HLA federations, which may 
include simulations widely, distributed over 
space, will exist where appropriate, but these 
federations should be standardized and validated 


for consistency across federates before use. The 
creation of “on the fly” federations that grab 
objects and federates scattered across the DII to 
create unique environments is probably still not 
possible in this end state because there is no 
guarantee that the representations will be 
internally consistent (consistency is not assured 
even if data is passed through HLA), and it is 
unlikely that a single analyst or team located at a 
single site will be familiar with all of the objects 
or federates potentially available. Instead, these 
simulations may be combined through the use of 
collaborative analysis rather than software 
linkages. 

All of the tools are presumed to be in a bed of 
warm data. That is, there is a data infrastructure 
that makes data readily available (with 
exhaustive metadata explaining the derivatiofi 
and accredited use of the data) for all of these 
systems. For example, an analysis of real-world 
capability should be able to draw on the same 
data set showing current status and deployments 
of units that a real-time command and control 
system might use. Similarly, an analysis of a 
future capability (for example, a 2010 force in a 
particular scenario) should be able to draw on a 
validated and commonly used set of data to 
represent the scenario, system performance, 
OPFOR performance, etc. Generation and 
storage of this data requires a significant 
resource conunitment within DoD, but it is 
necessary to achieve the desired end state. 

A key feature of this environment is that the 
output of the models, as well as the results of 
specific analyses and the lessons learned from 
training and exercises are also added to the 
common database. This provides a useful 
source of information that can be of particular 
help to quick turn-around analyses, which may 
synthesize the outputs of previous models, 
exercises and analyses. 

The Action Plan 

The Synthesis Group identified a number of 
actions that should become part of a long-range 
action plan that the community should consider 
and flesh out over the next few years as it 
evolves the practice of military analysis in DoD. 
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There are several steps that MORS itself can 
implement immediately that have little work 
impact but enormous potential to improve the 
ability of this community to connect with each 
other and better meet the over all needs of its 
membership. In addition to the items listed 
below, the working groups provided a number of 
general and specific steps that will help the 
community learn to harness knowledge 
management and collaboration as well as to 
address emerging issues in the changing security 
environment. Generalized recommendations 
endorsed by the workshop include working the 
people, process and technology problems in 
order to ensure a better final product. 

Actions for MORS include: 

• Near Term: 

Update the MORS membership directory 
and database with more information on each 
member. 

Evolve the role of the Electronic Media 
Committee to address enterprise knowledge 
management. 

Contribute to a Code of Best Practices for 
Analysis and Joint Experimentation. 

• Mid Term: 

Undertake focused workshop activity on this 
topic. 

• Far Term: 

Use the Education Colloquia to address 
ideas that Knowledge Management is an 
important element of Professional 
Development 

Actions for DoD include: 

The working group recommendations 
provide a phased approach to knowledge 
management and collaboration 
implementation by focusing on people, 
process, technology and product, as well as 
list challenges and promising tools and 
methodologies to meet those challenges. 

Table 4 summarizes the focus and potential 
impact of the specific actions recommended. 


_ Summary Recommendations _ 

Vision/Policy: Develop and promulgate an Enterprise 

Vision and strategic directions. _ 

Culture: Take action to remove deeply rooted cultural 
barriers (e.g., remove the incentives to hoard 

knowledge) _ 

Organization: Ensure that there is buy-in by Top 
Management. Investigate organizational constructs 
including business rules that best enable global 

collaboration of military analysts. _ 

People: Provide incentives to analysts to collaborate 

and share knowledge. _ 

Process: Institutionalize processes to enhanee 
collaboration and the sharing of tacit and explicit 

knowledge among the analytical community. _ 

Analytic Tools: _ 

• Develop and emphasize joint standards in 

algorithm and data development. _ 

• Expand the development and maintenance of data 

warehouses. _ 

• Focus basic research and tool development to 

meet emerging challenges _ 

Table 4. Summary Recommendations 

First, cultural changes are a necessary 
prerequisite to effective transformation of the 
analysis community (e.g., the need to dispel the 
myth that “knowledge is power”). Second are 
several organizational issues that must be 
addressed. One of these is to ensure top 
management buys in to this transformation. 
Another includes reassessing how security issues 
can be addressed in an information-centric, open 
environment. In the third area, decision makers 
must recognize that the analysts involved in all 
phases of the assessment process constitute a 
critical intellectual resource. 

They must be given incentives to stimulate them 
to contribute and draw from future knowledge 
warehouses. Tying all this together are the links 
between the vision and the high level guidance 
to the analysis process and the primary products 
of that process (i.e., effective, timely, rigorous 
analyses). To implement this linkage, a set of 
enhanced processes is needed (e.g., 
institutionalized processes to diffuse tacit and 
explicit knowledge) supported by a broad set of 
tools (e.g., knowledge warehouses, collaboration 
tools). These processes and tools are constrained 
by available resources (e.g., funds, manpower) 
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such that a special funding mechanism may be 
required to satisfy time-urgent joint needs. In 
particular, basic research must be continued in 
those areas where understanding of 
phenomenology is not available (human decision 
processes, Information Operations/Information 
Warfare (lO/IW), etc) as well as the continuing 
development of analysis tools to address the 
emerging challenges. 

The Synthesis Group feels that by addressing the 
needs of the community to implement our 
vision, the practice of analysis can evolve from 
its current position of a limited toolkit and 
methodologies to address a limited number of 
issues to a complete tool set connecting analysts 
and decision makers to address the full array of 
national security challenges. 

This workshop has demonstrated that there is 
great potential to be realized from the harnessing 
of the knowledge in our heads and in the 
systems that support our analytical efforts. In 
addition, there is great power in undertaking a 
more collaborative approach to the analyses that 
we perform. The combination of enterprise 
knowledge management and collaboration can 
deliver better analysis products faster. 


Summary 

The way ahead to realize this vision will be 
difficult. Nothing less than a cultural change is 
required. The demands of the national security 
environment require that we find ways to alter 
the information security environment to permit 
knowledge sharing and collaboration on an 
enterprise scale. Getting analysts to collaborate 
and breaking down the barriers to that 
collaboration and knowledge sharing are 
fundamental changes that require bottom up as 
well as top down change. 

The members of this workshop acknowledge the 
potential value of such change. It is now up to 
us to obtain the buy-in of the DoD leadership by 
convincing them that the benefits are worth the 
investment of time, money and energy to realize 
the vision articulated by this workshop. 
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Appendix A: Acronyms 


ACATI 

Acquisition Category I 

AFSAA 

Air Force Studies and Analysis 

AoA 

Analysis of Alternatives 

ARL 

Army Research Laboratory 

ASD(C3I) 

Assistant Secretary of Defense for Command, Control, 

ASD(S&TR) 

Communications and Intelligence 

Assistant Secretary of Defense for Strategy and Threat Reduction 

C4I 

Command, Control, Communications, Computers and Intelligence 

C4ISR 

Command, Control, Communications, Computers, Intelligence, 

CAA 

Surveillance and Reconnaissance 

Center for Army Analysis 

CEO 

Chief Executive Officer 

CINC 

Commander in Chief 

CNO 

Chief of Naval Operations 

CONORS 

Concepts of Operations 

COTS 

Commercial-Off-The-shelf 

CRAM 
on COE 

Defense Information Infrastructure Common Operating Environment 

DMDC 

DMSO 

Defense Modeling and Simulation Office 

DUSD (R&R) 

EUCOM 

US European Command 

EXCMS 

Executive Committee for Modeling and Simulation 

FEE 

Fleet Battle Experimentation Rrogram 

FFRDC 

Federally Funded Research and Development Center 

GIG 

Global Information Grid 

HLA 

High Level Architecture 

10 

Information Operations 

ISR 

Intelligence, Surveillance and Reconnaissance 

IW 

Information Warfare 

IWAR/CRAM 

IWARS 

IWEDA 

Integrated Weather Effects Decision Aid 

J-8 

Director, Force Structure Resource and Assessment 

JDS 

Joint Data Support 

JFCOM 

US Joint Forces Command 

JMEM 

Joint Munitions Effectiveness Manual 

JSF 

Joint Strike Fighter 

JTCG 

Joint Technical Coordination Group 

JTF 

Joint Task Force 

KM 

Knowledge Management 

M&S 

Modeling and Simulation 

MEDEVAC 

Medical Evacuation 

MOOTW 

Military Operations Other Than War 

MORS 

Military Operations Research Society 

MRS-05 

Mobility Requirements Study - 2005 

MSEA 

Modeling and Simulation Executive Agents 

MSRR 

Modeling and Simulation Resource Repository 
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NATO 

North Atlantic Treaty Organization 

NMCC 

National Military Command Center 

NTC 

OA 

National Training Center 

OASD(C3I) 

Office of the Assistant Secretary of Defense for Command, Control, 
Communications and Intelligence 

ODPA&E 

Office of the Director for Program Analysis and Evaluation 

OMFTS 

Operational Maneuver From The Sea 

OPFOR 

Opposition Force 

OPLAN 

Operations Plan 

OPSEC 

Operations Security 

OPTEMPO 

Operations Tempo 

OSD (PA&E) 

Office of the Secretary of Defense (Program Analysis and 
Evaluation) 

PACOM 

US Pacific Command 

PERSTEMPO 

Personnel Tempo 

POM 

Program Objective Memorandum 

PPBS 

Planning, Programming and Budgeting System 

PTD 

Phased Threat Distribution 

QDR 

Quadrennial Defense Review 

SECNAV 

Secretary of the Navy 

SrPRNET 

SECRET Internet Protocol Router Network 

SME 

Subject Matter Expert 

T&E 

Test and Evaluation 

TPFDD 

Time-Phased Force Deployment Data 

TRANSCOM 

US Transportation Command 

WG 

Working Group 
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Appendix B: Working Group and Synthesis Group Reports 


Working Group 1: Management of Analytic Knowledge 
Working Group 2: Enabling Collaboration 
Working Group 3: New Analysis Challenges 
Working Group 4: New Analysis Tools and Methods 
Synthesis Group 
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Working Group #1 Members 


Mr. Michael Yoemans 
Mr. Gary Coe 
Mr. John Black 
Ms. Joan Conover 
Ms. Mary Margaret Evans 
ETC Jim Ferguson 
Mr. Robert Graebener 
Mr. Ernie Gurany 
Ms. Carol Huber 
Mr. Douglas Martin 
Dr. Christopher Pernin 
Dr. Edward Schmitz 
Dr. Jeffery Schofield 
Mr. Jack Sheehan 



OASD(C3I) Chair 

IDA Co-Chair 

JFCOM 
JFCOM 

OSD Reform Initiative Office 

Joint Staff/J-8 

IDA 

NGIC 

USAISSC 

Booze, Allen, and Hamilton 
RAND 

Dept of Navy, CIO 

IDA 

DMSO 
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Working Group 1 Presentations 


♦ Mr. Michael Yoemans, Introduction to Knowledge Management 

♦ Mr. Gary Coe, The Analytic Enterprise 

♦ Mr. Robert Graebener, KM Lessons from STOW 

♦ Mr. John Black, A Vision of an Analytic Portal 

♦ Mr. Jack Sheehan, DMSO Initiatives to Identify, Access and Store 
Knowledge 







WG #1 Management of Analytic Knowledge 
Focus Questions 

♦ How is analytic knowledge currently discovered, captured, 
expanded, organized, and shared within the DoD analysis 
community? 

♦ What Knowledge Management techniques can be applied to 
improve DoD analysis? 

♦ What analytic processes would be impacted by the 
implementation of Knowledge Management? 

♦ What are the obstacles in successfully implementing 
Knowledge Management to the DoD Practice of Analysis? 

♦ What recommendations do you have to improve the existing 
analytic practice using Knowledge Management? 

G5S 
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WG #1 Management of Analytic Knowledge 

The Workgroup’s Vision 


Informed Decisions resulting from effective 
and efficient analyses enabled by the sharing, 
collaboration and creation of knowledge. 








WG #1 Management of Analytic Knowledge 

The Strategy 


♦ DoD Leadership by J-8 and OASD(C3I) to establish policies 
and facilitate change. 

♦ Change the mindset of senior leaders and culture in DoD to 
promote collaboration and knowledge sharing. 

♦ Partnering of appropriate DoD organizations and agencies to 
overcome the barriers to knowledge management 
implementation. 

♦ Continuing education and training in knowledge management 
practices and technologies. 

♦ Prototyping of knowledge management implementation to 
important problems to evolve solutions. 
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WG #1 Management of Analytic Knowledge 

Current Use of Analytic Knowledge in DoD Analysis Community 
“SILOED” KNOWLEDGE 

♦ Information “silos” determined by organizational boundaries 
—division, functions, etc. 

♦ Studies centered. 

♦ Navigation via operating and file systems. 

♦ Reliant on internally generated information. 

♦ Information shared via email attachment. 



Current analysis is done in-house using information available inside the 
organization. Existing information sources are designed to meet functionally 
oriented information needs — not for doing operational analysis. 
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WG #1 Management of Analytic Knowledge 

Knowledge Management Techniques (Page 1 of 3) 


“SILOED” KNOWLEDGE 

♦ formation “silos” determined 
by organizational boundaries—^ 
division, functions, etc. 

♦ Studies ceiitered.,' 

♦ Navigation via operating and 
file systems.. - - 

♦ Reliant oh: internally generated 
information. '■ 

♦ Information sh^ed via email 
attachment. 


GS3 


ENTERPRISE KNOWLEDGE 

♦ Enterprise-wide integration of 
information resources. 

♦ Issues centered (Information, 
people and tools). 

♦ Navigation via hypertext 
browser links and search 
engines. 

♦ Integrates external information 
from strategic partners, 
suppliers, customers and third 
party sources. 

♦ Information shared via 
knowledge portals. 


As contrasted by the status quo in a knowledge organization analysts would use, 
or have access to, enterprise-wide knowledge repositories and should be issues 
oriented, leveraging the evolving technologies that permits board navigation of 
the knowledge-base. 
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WG #1 Management of Analytic Knowledge 

Knowledge Management Techniques (Page 2 of 3) 

Private Sector 

♦ THESAURAL BROWSERS 

Browsers generate lists of related terms and resources, allowing 
users to take advantage of otherwise unexpected associations 
between seemingly unrelated information sources and types. 

♦ PERSONALIZED INTRANET PORTALS 

The creation of customizable intranet interface allows analysts 
selective access to information resources required to perform their 
jobs, enables employee automation of simple, routine tasks and 
creates a vehicle for a central information-management body to 
“push” relevant content to distributed users in real time. 

♦ KNOWLEDGE SHARING INCLUSIVE PERFORMANCE 
REVIEWS 

Incorporation of a knowledge-sharing into the corporate culture 
with incentives. 

d&S _ 


Industry has provided the models that DoD can leverage. A study recently done 
by the Working Council for the Chief Information Officers demonstrates this 
point. WG #1 included a copy of this material as a part of their proceedings. 
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WG #1 Management of Analytic Knowledge 

Knowledge Management Techniques (Page 3 of 3) 

Private Sector 

♦ FEDERATED KNOWLEDGE MANAGEMENT 

Best-practice teams in analysis divisions or analysis units leverage 
their subject expertise to discover and codify best practices, which 
are then passed on to a corporate knowledge-management group 
that facilitates the propagation of best practices across the 
enterprise. 

♦ INFORMATION-EXPIRATION MECHANISMS 

Intranet content is automatically identified by date of creation 
and/or last modification; dated documents are flagged and routed 
to their authors, who review them for relevancy and decide to 
archive, delete, modify or reinsert the documents back into the 
corporate database. 

♦ CONTENT-VALIDATION PROTOCOLS 

A process to prevent the propagation of inconsistent or incorrect 
information throughout the enterprise. 
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WG #1 Management of Analytic Knowledge 

Analytic Practices That Use Analytic Knowledge 


The DoD Analytic Process 

Skilled Labeled by Products 


Analysts 


ISSUES 


MEASURES 


Data Sources 
and Validation 


STUDY 

PLAN 


STUDY 

EXECUTION 


DATA 
ARCHIVE & 
RETRIVAL 


Instrumentation 


DISSEMINATION OF 
REPORT WITH 
CONCLUSIONS and 
RECOMMENDATIONS 


Plan for Data 



Collection 


In order to implement Knowledge Management (KM) to the DoD Analytic 
Process, we need to have a common understanding of this complex process. 
This was the working groups (1&2) consensus and all agreed that such an 
understanding was critical to successful application of KM practices. 
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WG #1 Management of Analytic Knowledge 

Obstacles to Implementing Knowledge Management 

♦ Policy makes it impossible to use existing knowledge tools. 

♦ There is no Global Information Grid. 

♦ The pace of technological change is high. 

♦ No economic model that provides the incentive for sharing. 

♦ Organizational and individual resistance. 

- Resistance to change. 

- What’s in it for me? 

- Is this “too hard”? 

- The institutional incentive to hoard information and knowledge. 

♦ The Challenges of Managing Unstructured Information (e.g. 
analyses). 

♦ The need for security and protection of information in the 
conduct of many analyses. 

- Th e value of intellectual property. 

SB 


The working group envisioned a number of major obstacles that would need to 
be overcome in order to implement KM successfully throughout the analytic 
community. This slide shows the important problem areas that the working 
group tried to address in their recommendations. These obstacles were carefully 
considered in the formulation of the strategy that the working group 
recommended in implementing KM to the analytical community. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 1 of 7) 

♦ The DoD Sponsor implement knowledge management techniques, 
particularly to establish an intranet portal, in support of an 
important study with suitable scope, schedule, and content that 
partners DoD components Military Operations Research Society, 
industry, and academia. The elements that should be considered 
along with strategy already discussed include: 

- People 

- Process 

- Technology 



This is our overarching recommendation, which detailed in the next series of 
slides under the headings of: 

Strategy 

People 

Process 

Technology 

Specific organizations that should be included are the Services, OASD(C3I), 
DTIC, MSIAC, DMDC and DMSO. 
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WG #1 Management of Analytic Knowledge 

Recommendations Page 2 of 7 


♦ People 

- DoD and MORS lead an assessment of the capability to provide a 
baseline inventory of analysts for DoD operations research studies 

- MORS assess its capability to establish a Knowledge Management 
Professional Development Program for the DoD Analytic 
Enterprise that would include the identification of skills and 
products, the refinement of skill levels, and the training of 
analysts. 

~ The DoD sponsor evaluate successful skill-inventory systems (e.g. 
Northrop-Grumman and Y2K) for implementation within the DoD 
Analytic Enterprise. 



Industry experience and DoD case studies show that people is one of the most 
critical success factors associated with KM. The recommendations cited here 
are designed to provide the fundamental foundation for organizing and 
educating people using KM techniques being applied by industry. Key DoD 
organizations that could be involved in pursing this recommendation include 
ASD (C3I), DUSD(P&R), OSD(PA&E), and DMDC. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 3 of 7) 


♦ Process: 


Skilled 

Analysts 


J-8 accept leadership with OASD(C3I) of an evolving analysis 
process for the purpose of improving analysis in a future analysis 
state. Notionally, the evolving process could be closely interfaced 
with an evolving Knowledge Management Process as shown on 
the next slide. 


Instrumentation 
Plan for Data 
Collection 
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WG #1 Management of Analytic Knowledge 

Recommendations (Pagq^4 of 7) 


Report* 


decisions' 
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The working group recognized that the future (to-be) process incorporating KM 
is still to be defined. Nevertheless, this slide helps illustrate the important 
interrelationships related to collaboration, creation, and reuse of a well 
constructed analytical KM environment and extensible knowledge-base. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 5 of 7) 


♦ Process continued: 

- The DoD determine the business rules for contributing analytic 
knowledge. For example, what are the current rules that control 
the access to past studies and how could these rules be changed to 
improve access such as those stored at Federally Funded Research 
and Development Centers (FFRDC). A special case is the 
identification of the business rules for administering and 
controlling an analyst’s portal. 

- The DoD Sponsor, with the assistance of MORS, identify new 
types of information required for future analyses (e.g. foreign 
demographics, information warfare related, economic information, 
national infrastructure related, chemical/biological, etc). 

- The DoD Sponsor identify current Centers of Expertise required 
for improved analyses. 


SB 


The working group concluded that the first bullet was probably the most critical 
follow-on task for J8 and ASD (C3I) to tackle. If this is not done KM will fail. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 6 of 7) 


♦ Process continued: 

- Take steps to systematize the organization of information by 
transposing principles from traditional library and information 
sciences to the electronic realm. 

- Develop and use an Information Architecture that structures the 
design of information organization, labeling, navigation, and 
indexing systems to support both browsing and searching. 
Identify and use a set of metrics for the business case preparation 
and performance evaluation of the Analyst’s Portal. 

*- Create Knowledge-Sharing Incentives such as: 

» Knowledge-Sharing inclusive Performance Reviews 
» Embedded Authorial Recognition 

OS 


The importance of performance reviews and embedded are mentioned 
repeatedly in the literature. The study done by the Working Council for the 
Chief Information Officers strongly supports this point. This study also stressed 
the importance of having an information architecture for effective design and 
administration of browser-enabled knowledge-based repositories. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 7 of 7) 


♦ Technology 

- The DoD concentrate on applying COTS tools as a key critical 
success factor for building analytical KM portal. Example 
technologies include: 

» Thesaural browsers 
» Personalized Intranet Portal 
» Knowledge-Sharing Inclusive Performance Reviews 
» Federated Knowledge Management 
» Information-Expiration Mechanisms 



WGl-19 







WG 1 - 20 



MORS Workshop 


.••. 'T- • .• 


Evolving the Practice of Military 
Operations Analysis in 




Working Group #2 
Enabling Collaboration 




Co-Chairs: 




COL Robert D. Clemence, Jr. 

. 



February 29, 2000 through March 2, 2000 
Naval Postgraduate School, Monterey, CA 












WG 2 Participants 

COL Bob Clemence 

JCS/J-8 Chair 

LtCol Kirk Yost 

JCS/J-8 Co-Chair 

Dr. Tom Allen 

IDA 

Ms. Bobbin Beall 

CNO/N8I6 

LTC Jerry Glasow 

DUSA(OR) 

Dr. Dan Fox 

RAND 

Mr. John Bordeaux 

ITT Research 

Maj Dan Zalewski 

OSD/PA&E 

Dr. Richard Ryberg 

LOGICON 

Mr. Jay Wilmcth 

LOGICON 

Ms. Annie Patenaude 

SAIC 

MAJ Sue Romans 

CAA 

Dr. Louis Moore 

RAND 

Ms. Elaine Simmons 

JDS 

Maiunv opwiioss wsruicH socieit 



Fourteen experienced analysts from government, the FFRDCs, and industry 
participated in the eight working group sessions and should be given the credit 
for the contributions I will present. 


For our purposes, we have defined collaborative analysis to be any analytic 
activity that involves the sharing of data, tools and expertise across 
organizational boundaries to achieve a common goal. 
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WG 2 Presentations 


♦ COL Bob Clemence, JCS/J-8, Joint Analysis: A Status Report 

♦ LtCol Kirk Yost, JCS/J-8, Organizational Alternatives for 
Collaborative Analysis 

♦ COL Jim Methered, EUCOM, The Future of Joint Analysis: An 
Empirical Look for Lessons 

♦ LTC Jerry Glasow, DUSA(OR), Input-Based vs. Output-Based 
Collaborative Analysis 

♦ MAJ Sue Romans, CAA, Real Time Analysis 

♦ Ms. Elaine Simmons, JDS, Resolving Data Issues in 
Collaborative Analysis 

QS 


We used contributed papers to motivate discussions on issues that individuals 
and organizations must confront before, during and after collaborative work, 
such as: 


•When is collaboration useful? 

•How should collaborative effort be organized? 

•If information is power, how readily will organizations share their data? 
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WG #2 Enabling Collaboration 
Focus Questions 

♦ How is collaboration currently used in the DoD analysis 
community? 

♦ What formal and informal collaboration techniques can be 
applied to improve DoD analysis? 

♦ What analytic practices (local study, centrally-managed 
study, etc.) would make use of collaboration support? 

♦ What recommendations do you have to improve the existing 
analytic practice using collaboration? 



Four focus questions were proposed before the Workshop to guide discussions. 
In our deliberations, we could not make a distinction between questions #1 and 
#3. The remainder of this presentation will address questions #1, 2 and 4. 
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WG #2 Enabling Collaboration 

Current Use of Collaboration in DoD Analysis Community 

♦ CINC analyses to support current operations: 

- Collaborative by nature but much more complex than in past. 

- National, International and Joint considerations. 

- Compressed decision cycle - come as you are (HW, SW, data, skill-set). 

- Event driven and spontaneous. 

- Limited by lack of forward analyst presence (few collaborators available). 

- Frequent requirement for Deployment and Logistical Analysis. 

♦ DoD level analyses to answer congressional language or support 
PPBS: 

- Collaboration is directed. 

- Joint and Service considerations pre-dominate. 

- Results important, but not urgent - time to acquire tools, data and skills. 

- Formal process - Terms of Reference, Study Plan, Oversight. 



We have identified three archetype analytic activities where collaborative 
analysis takes place. 

Analysis in the unified commands is collaborative by nature due to the military, 
political, inter-agency, and international considerations. In this real-time 
environment, the decision-cycle is so compressed that analysis must be agile to 
be useful. If tools, data, and talent are not available when called, the decision 
will be made without analytical insight. Analysts in EUCOM have learned, for 
example, that they are frequently asked to provide deployment and logistical 
analysis. Collaborative work is made more difficult by geographic separation 
and OPSEC concerns. 

Within the Beltway, collaborative analysis is frequently directed to answer 
Congressional Language or to support PPBS decisions. In this environment, 
joint and service considerations pre-dominate. Unlike the environment 
previously described, these activities are important, but not so urgent that tools, 
data, and skills cannot be massed and employed in a deliberate, formal way. A 
distinguishing characteristic of this kind of collaborative work in a hierarchical 
oversight structure that guides the analysis from cradle to grave. 
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WG #2 Enabling Collaboration 

Current Use of Collaboration in DoD Analysis Community 


♦ Service analyses to support acquisition, PPBS and doctrine 
development: 

- Collaborative when analysis is set in a Joint context. 

- Service considerations predominate. 

- Collaboration may be formal or informal. 



The third collaborative analysis archetype is the service/agency study performed 
to meet an internal requirement. Intra-service and inter-service collaboration is 
driven by the needs of the service customer. Analyses of Alternatives (AoAs), 
for example, for Acquisition Category I (ACAT-I) systems frequently require a 
joint context. Consultation and rendered assistance across organizational 
boundaries may be formal (by memorandum) or informal (by professional 
courtesy). 
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WG #2 Enabling Collaboration 

Formal and Informal Techniques That Enable Collaboration 

♦ Direct Analytical Support: 

~ Example: Deployment from CAA to EUCOM to support exercise. 

- Not responsive enough for crisis action planning. 

- Working relationships (organizational, personal) established in 
advance and rehearsed. 

- Co-location is preferred to tele-presence. 

♦ Data Warehousing: 

- Trusted agent for data verification, validation and archival. 

- Adherence to formal data access and release procedures. 

♦ Trust-Building Activities: 

~ Participation in MORS working groups, workshops, etc. 

- Positive experiences with earlier collaborative work. 



Collaboration can be encouraged and assisted by both formal and informal 
mechanisms. One motivation for an organization to collaborate is acquire the 
use of tools and expertise to meet infrequent requirements (e.g., support to Joint 
exercises or contingency plan development). EUCOM and Center for Army 
Analysis believe that this arrangement works best when working relationships 
have been established in advance and are rehearsed. In this way, the supporting 
organization understands the decision making processes of the supported 
organization and the hardware/software/security requirements for 
interoperability. While collaborative work can be done from a distance, 
EUCOM prefers on-site support. Co-location helps allay fears concerning 
operations security, responsiveness and miscommunication. 

The existence of an “honest broker” for data encourages participants to share 
their data holdings. The Joint Data Support (JDS) Office has earned the trust of 
the Services, the Joint Staff, and other offices in the Secretariat by the way it has 
controlled the data entrusted to it during Mobility Requirements Study 2005 
(MRS-05). Perhaps more important than its role as a “gatekeeper” is the service 
JDS provides to the community by cross-checking the data it receives against 
other authoritative sources to identify data anomalies. These anomalies are 
provided back to the source for resolution, improving the veracity and 
consistency of DoD data holdings. 

Informal trust-building activities like MORS and positive experiences with 
collaborative analysis help to overcome organizational and individual resistance 
to teaming. 
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WG #2 Enabling Collaboration 

Observations on Collaborative Analysis 

♦ Collaboration only works when people and 
organizations are willing to set aside parochial (or 
cultural) behaviors for a higher purpose: 

- Common Goals Do Not Guarantee Consensus. 

- Achieving consensus does not encourage bold shifts. 

♦ Paradigms for collaborative studies achieve different 
outcomes 

centralized (input-based) vs distributed (output-based): 

- Centralized - control the inputs and models to produce one, accredited 
result. 

» Susceptible to single-point failure; deterministic approaches produce 
point estimates. 

- Distributed - encourage independent efforts using the same assumptions 
to produce a distribution of results for executive consideration. 

» More robust, but any participant can warp the process by submitting 
biased input. 

» Requires reconciliation of independent results. 



It is important to remember the collaborative analysis is not an appropriate 
technique for all situations. A quote attributed to Napoleon is “I would rather 
fight allies than be one.” Sharing common ends does not guarantee agreement 
on the ways and means used to get there. Group pressure to cooperate can result 
in compromises that preserve the status quo when what was really needed was 
creativity and a bold shift. 

LTC Jerry Glasow presented a paper in our working group that classifies 
collaborative methods as being either “input-based” or “output-based.” MRS- 
05 typifies the former while the Phased Threat Distribution (PTD) Study is an 
example of the later. The “input-based” approach reasons that if the participants 
accredit (1) the data, (2) the tools and (3) the analysts, the results will, in turn, 
be accredited. When this approach is implemented by a sequence of models 
(e.g., air defense precedes deployment precedes force-on-force evaluation) a 
single point failure can delay the entire project. It also tends to produce point 
estimates rather than a range of possible outcomes for consideration. 

Two advantages of the “output-based” approach are parallel, independent 
analysis and multiple results that can be compared. The most difficult part of 
this approach can be determining why the results are different. If this approach 
is used, several integrated teams should be commissioned by the participants 
rather than putting the participants in competition with one another. 
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WG #2 Enabling Collaboration 

Observations on Data Collection and Production 

♦ Data collection, production and preservation need to be integrated 
into day-to-day operations: 

*“ Need to address multi level security issues. 

- Need to capture operational data at JTF level while data is still purple. 

- Need to plan for data capture. 

♦ Collaboration slows as data collection and production burden 
increases: 

- Deluge of requests during operations. 

- OPSEC issues. 

♦ Must understand limits of data: 

- Close-hold / incomplete data. 

- Data collection, production, preservation are manpower limited. 

♦ Documentation critical to accrediting data for collaborative use 

- Includes metadata (technical, informational, operational). 


Data collection, production, preservation and maintenance was a recurring 
theme in our deliberations. Jim Methered, citing his experience during and after 
OPERATION ALLIED FORCE, is vehement that data collection must be a 
component of the JTF information management plan instead of the “ad hocery” 
that exists today: 

“...scavengers picking over small, rapidly decaying, incomplete, and 
questionable data and then fiercely defending the scraps gathered from others.” 

A potential major future role for operations analysis may be to study how 
information is used and develop ways of managing and using it better. Process 
engineering in peace might facilitate analysis in war as well as improve actual 
and simulated decision making processes. 
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WG #2 Enabling Collaboration 

Recommendations 

♦ People 

- Find ways to identify (and evaluate?) Subject Matter Experts (SMEs). 

“ Find a means to develop operational experience in Civil Service 1515s 
and uniformed analysts w/o warfighting specialties. 

♦ Process 

- There is a spectrum of collaborative analysis processes, and each has 
its strengths and weaknesses (bold shifts versus consensus). 

» Can organize for competitive analyses. 

» Can organize to search for a “common ground” solution. 

- Collaborative Analysis is a “teamsport” and requires practice. 

» OA support to commanders in the field most effective when “train 
as you fight” has been implemented. 

- Reestablish formal liaison/exchanges to improve the quality of 
an alysis. 

OS 


Our recommendations have been collected under the headings of “People,” 
“Process” and “Product.” 


Collaboration requires colleagues with whom to collaborate. We believe there 
should be a better way of identifying the experts in our community than word of 
mouth. We also recognize that changes in the way the uniformed services are 
managing the careers of military analysts may make collaborative work more 
essential in the future. We must either find a way to increase the “field 
experience” of our analysts or find ways to include “operators” in our analysis 
teams. 


Collaborative analysis takes longer to perform than analysis done in isolation 
Timelines can be shortened if informational exchanges and liaisons between 
analytic organizations/agencies are institutionalized in our business practice. 
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WG #2 Enabling Collaboration 

Recommendations 

♦ Process (Cont’d) 

- Support independent, accredited sources for data to speed collaborative 
effort. 

- Find ways to identify data to source new applications and phenomena. 

- Data collection must be deliberate and done in such a way that it does 
not require additional effort by operators. 

- Synchronize data production with data demands and institutionalize 
production and update of long-lead items. 

♦ E.g., extend JMEM to include developmental systems. 

- Harmonize schedule of major collaborative projects and Service 
workload. 



In the short-term, its important that we reconcile our demands for data with our 
ability to produce it. Dr. Chris Lamb, ASD (Strategy and Threat Reduction) is 
currently leading an effort with the services and the intelligence agencies to 
better coordinate our efforts. It also makes good sense to plan in advance for 
the long-lead data that are common to many of the studies we do (e.g.. Time- 
phased Force Deployment Data or TPFDDs) In the long-term, data collection in 
the operational environment needs to evolve from something that interferes with 
task performance to something that is transparent to people performing their 
day-to-day tasks. Similarly, directives to perform large, collaborative analysis 
needs to be harmonized with the day-to-day analytic workload. 
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WG #2 Enabling Collaboration 

Recommendations 


♦ Product 

- Need improvements in literature search capabilities, identifying 
work in progress that may affect the project at hand. 

~ Study documentation techniques must also improve, allowing for 
reproducibility of results and reuse of data in addition to 
description of procedure and findings. 



Compressed decision-cycles and the tendency toward using briefings rather than 
reports as deliverables has lessened our use of literature search and our 
discipline to thoroughly document what we do. 
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Working Group #3 Members 
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Working Group #3 Presentations 


♦ Dr. Jackie Henningsen, Air Force Analysis in the Context of 
Today’s Expeditionary Aerospace Force 

♦ Capt Russ Bergeman, Analytic Challenges of the USMC 

♦ Mr. Chuck Werchado, OPNAV Naval Analysis: Recent Navy PPBS 
Process Changes and QDR Analysis Initiatives 

♦ LTC Jeff Appleget, Challenges in Army Modeling 

♦ Mr. Mike McCurdy, Analysis Challenges for Joint Operations - A 
USCINCPAC Perspective 
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WG #3 New Analysis Challenges 

Applicable Workshop Goals and Objectives 


♦ Questions 

- What problems are not well served by the current state of the 
practice? 

♦ Goals 

- Develop a shared understanding of the need to evolve the practice 
of Military Operations Analysis in DoD. 

- Establish a vision for improving analytic practice. 

- Objectives. 

- Identify weaknesses in the current practice relative to current and 
future requirements. 


The objective of Working Group #3, New Analytical Challenges, was to address 
the weaknesses in the current practice of analysis within the Department of 
Defense. 

The working group focused on giving each of the services and the Joint Staff the 
opportunity of presenting their current analytical challenges.. .problems that 
they are facing that cannot be solved with the current analytical toolset. The 
working group’s goal was to define the areas that need to be evolved within the 
practice of DoD analysis. 
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WG #3 New Analysis Challenges 

Focus Question 


♦ What are military operations analysis questions of interest for 
which the established military operations analysis community 
apparently does not have answers? 

- i.e., What problems are not well served by the current state of the 
practice? 



The focus of Working Group #3 was to examine military operations analysis 
questions of interest for which the established military operations analysis 
community apparently does not have the answers. This working group is about 
new problems, problems not involving conventional warfare, and/or post-Cold 
War problems. Actually there are old problems that belong here as well. The 
issue isn’t whether the problem is new or old, but rather whether the Military OR 
conununity has succeeded in addressing it satisfactorily. An example is the 
problem of determining the value of a common operational picture to the 
warfighter. Presentations to this working group came from the major analysis 
players such as the Services’ Studies and Analysis shops, from warfighters like 
the Joint Forces Command and USCINCPAC, and from decision makers like 
OSD, J8, and the Services' warfare requirements and assessment organizations. 
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WG #3 New Analysis Challenges 

Problems not well served by the current state of MOR practice 


♦ Methods for Analyzing New Domains of Operation 

- High intensity/low casualty conflict 

- MOOTW, humanitarian operations, peacetime engagement, 
operations in urban terrain 

- Technology insertion (digital battlefield, tomorrow’s dismounted 
warrior) 

~ Chemical/biological warfare effects (including contagion) 

- Asymmetric threats (terrorism, political manipulation, homeland 
defense) 

» Means for recognizing attack 

- Application and effects of special operations 

- Information operations/warfare 

QSg 


With the end of the Cold War, the Department of Defense and thus the DoD 
analytical community, are faced with the demand of conducting and analyzing a 
variety of new operations. These new operations require new methods of 
analysis for planning, conceiving concepts of operations, defining doctrine and 
analyzing effectiveness. Some examples of these new domains are: High 
Intensity/Low Casualty Conflict, Military Operations Other Than War 
(MOOTW), humanitarian operations, peacetime engagement, operations in 
urban terrain technology insertion (digital battlefield, tomorrow’s dismounted 
warrior), chemical/biological warfare (including contagion), asymmetric threats 
(domestic terrorism, political manipulation, homeland defense), special 
operations and information operations/warfare. 
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WG #3 New Analysis Challenges 

Problems not well served by the current state of MOR practice 


♦ Model-centric focus limiting analysis toolkit 

- Scenario development and data collection time consuming 

- More scoping analysis to set stage for detailed study 

- Encourage exchange of simple analysis tools (e.g., spreadsheets) 

- Combine results from all pertinent tools 

♦ Identifying complex interrelationships between factors 

- communicate effects of multi-dimensional trade space 



The current emphasis on modeling and simulation with it’s resulting de¬ 
emphasis on other analytical tools and methods is limiting the analysis toolkit. 
With the amount of time and resources required for scenario development and 
data collection to run the current models and simulations, some other 
alternatives are clearly needed. For example, methods of scoping analysis that 
can quickly look at the alternatives and potentially set the stage for further, 
larger, study are needed. These scoping tools can serve to pare down the 
alternatives and allow the focus on the important question to evolve. In 
addition, simpler analysis tools such as spreadsheets should be used and 
exchanged within the analytical community as a rapid means of knowledge 
transfer. Finally the analyst should be taught to combine the results from all 
pertinent tools and analytical methods to more thoroughly, accurately and 
quickly answer the question at hand. 

With the increasing complexity of today’s military scenarios and high 
technology weapons, there is a greater need for the analysis community to be 
able to identify and communicate the effects of the factors and their 
interrelationships. It is especially important to be able to conduct and 
communicate the trades conducted between factors that typically would not be 
linked. For example, an aircraft carrier can be the platform used for JSF 
launches, and a typical analysis may look at the number of targets put at risk by 
various sizes of JSF fleets. However, an interrelated measure that should be 
considered is the deck space that is being taken up by the JSF fleet, that may 
potentially be used for such things as helicopter medevac support., .two very 
important missions with interrelations that must be considered. 
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WG #3 New Analysis Challenges 

Problems not well served by the current state of MOR practice 


♦ Analytical methods that trace factors to operational and 
strategic outcomes: 

- Logistics 

- Bandwidth 

- Engagement/presence on PERSTEMPO and OPTEMPO 

- Infrastructure Investments 
“ Readiness postures 

- Force structure changes (deployable, lighter, lethal fighting force) 


OSS 


There are many factors that effect the outcome of a battle, but do not have 
directly measurable combat effects. These factors must be taken into account, 
and their effects must be traced to operational and strategic outcomes. These 
factors include such things as logistics, bandwidth requirements, 
engagement/presence on PERSTEMPO and OPTEMPO, infrastructure 
investments, readiness postures and various force structure alternatives. 
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WG #3 New Analysis Challenges 

Problems not well served by the current state of MOR practice 


♦ Methods for including behavior factors in current tools 

- Quantifying small unit behavior 

» Effect of situational awareness on force effectiveness 

- Quantifying adversary leadership motivation 

» Effect of target selection in influencing course of action 

♦ Methods of quantifying risk 

♦ Real time analysis for operational support 



The behavior of individuals, small units, and adversary leaders must be captured 
in our evolving analytical tools. The effect and measurement of behavior 
factors at the individual or small unit levels are important considerations, 
especially in the analysis and evaluation of combat systems. For example, the 
increase in situational awareness for the individual foot soldier is probably a 
good thing, to a point — until information overload occurs or if the equipment 
used to give the situational awareness decreases the individual’s effectiveness in 
other areas, such as mobility. On the other hand, being able to quantify the 
characteristics and motivations of an adversary leader would allow the 
identification of the leader’s or country’s centers of gravity and may assist the 
planning process in identifying effective target sets that would influence the 
enemy’s course of action. 

Methods for quantifying risks of all sorts must be developed and 
institutionalized. 

The frequency of calls for quick-tum around analysis is increasing. These 
analyses take many forms, such as support for testimony to Congress or support 
to ongoing warfighting efforts. To answer these types of questions, the analyst 
needs a set of tools that support these quick analysis activities. These tools need 
to capture the important elements that would be drawn out by more thorough 
analysis but run quickly enough to support answering the question when the 
answer will still be desired and relevant. 
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WG #3 New Analysis Challenges 

Recommendations 


♦ Find methods to deal with: 

- New forms of operations. 

Traceability of known factors and interactions. 

- Quick turn-around studies. 

♦ Nurture analysts and analytical thinking rather than focusing on 
model development: 

- Project more balanced view of model capabilities and limitations. 
-- Reverse trend to big models as solution to all problems. 

♦ Characterize and communicate impact of uncertainties (e.G., 
Risk, behavioral factors). 


In summary, as the analytical community evolves it needs to find tools to deal 
with new forms of operations, find methods to trace known factors and 
interactions to their logical combat output, and find tools that will provide 
accurate and thorough analysis in a short timeline. 

The DoD analytical community needs to nurture analysts and emphasize 
analytical thinking rather than focusing on simulation and modeling 
development. The community must project a more balanced view of model 
capabilities as well as limitations, and reverse the recent trend of blind faith 
toward large models as the solution to all analytical problems. 

The analytical community needs to find means for characterizing and 
communicating the impacts of uncertainty. For example risk, such as decision 
risk based on the fact that various factors may not have been considered in the 
current analysis, need to be characterized and communicated to the decision 
maker. 
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The following slides provide the notes taken during the presentations and the 
resulting discussions of the working group sessions. The problems identified in 
the previous slides were distilled from these session notes. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Methods for recognizing/analyzing unrestricted warfare 

~ New domains of warfare, warfare reborn 

- Terrorism, high intensity/low casualty conflict 

♦ Apply traditional OR methodologies, but consider methods for 
including behavior factors 

- Draw upon expertise of multi-disciplinary teams (behavioral 
scientists, psychologists, etc.) 

- Effect of target selection to force enemy capitulation 

♦ Need measures to relate warfare outcome of non^warfare factors 
and metrics for other operations 

~ Logistics, bandwidth 

- MOOTW, humanitarian operations 

OS 


During her keynote address, Dr. Ruth Davis introduced the concept of 
unrestricted warfare. With this concept, war has been reborn, focused on such 
activities as terrorism and high intensity/low casualty conflict. These types of 
warfare emphasize the need to analyze economic and police efforts, something 
that the current analytical community tools are ill prepared to handle. 

We need to recognize that the behavioral science fields, although not as 
measurable as other tools, provide useful information to policy decision makers. 
Part of OR analysis’ value to decision makers may be to use these sciences to 
identify alternatives or structure the decisions. For example, developing a 
model that relates an adversary leaders’ motivations and vulnerabilities, could 
potentially be used to identify the target set that would force that leader into 
capitulation. 

The use of measures that quantify warfare outcomes, such as attrition, are well 
known. However, there are many factors that support warfare operations that 
are not directly quantifiable with the current measures and analytical techniques. 
The community needs to identify metrics that can be used to quantify the value 
of these non-warfare factors, such as logistics and bandwidth use and other 
types of operations, such as MOOTW and humanitarian operations. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Communicate effects of multi-dimensional trade space: 

“ Different system solution options, multi service. 

♦ With changing CONOPS and doctrine, analysis methods and 
human factors assessments must catch up: 

- Information technology insertion (digital battlefield, tomorrow’s 
dismounted warrior). 

- Force structure changes (deployable, lighter, lethal fighting force). 
~ Operation in urban terrain. 




The analytical community needs to find new techniques that will allow the 
capability to communicate the effect of the factors within a multi-dimensional 
trade space. This trade space includes trading various system solutions or 
options offered across service boundaries. 

The new environment that includes rapidly advancing technology and the 
paradigm that “change is good” is rapidly changing CONcepts of OPerationS 
(CONOPS) and doctrine. While the way the military does business is rapidly 
changing, the environment is also providing new analytical challenges. We 
need to figure out how we fit into this new environment, and develop tools that 
will answer questions such as: 

- How to assess the benefits and drawbacks of information technology insertion, 
such as the digital battlefield and tomorrow’s dismounted warrior. For example, 
is there a point where too much information is too much for the warrior? 

- How to assess the benefits and implications of proposed force structure 
changes. 

- How to model operations in urban terrain, such as determining the level of 
detail of modeling buildings in an urban environment. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Challenges from analytical support to CINC’s 

~ Analysis tools and data for OOTW and peacetime engagement 
~ Indicators of inter/intra-statc instability 

- Real-time analytical support for operations and exercises 

» Short analysis time frame 

- Field analysis of operations and exercises 

- Combining training and analysis 

» Simulation support for mega-CAX’s 
» Combining analysis and exercise results 




From the CINC’s viewpoint, there are a variety of analysis challenges. 

Real time analytical support for operations offers a different environment, due 
to the very short timeframe allowed for the analytical activities. There is no 
time to build data bases, therefore the most valuable resource that the analyst 
can offer is an operational focus and a structure for thinking about the problem. 

Training simulations provide challenges in a few areas. Increasing the 
simulation support for combining many of the large exercises, and combining 
analysis and exercise results. 
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WG #3 New Analysis Challenges 

Session Summary 

♦ QDR analysis challenges: 

“ Methods of quantifying risk and uncertainty. 

- Feedback loops to include analysis in planning. 

- Analytical transparency. 

- Effects quantification: 

» Engagement/presence on PERSTEMPO and OPSTEMPO. 

» Infrastructure Investments. 

» Readiness Postures. 

» ChemicaiyBiological Warfare Effects. 

» Asymmetric Threats (terrorism, political manipulation). 

» Information OperationsAVarfare. 

- Compares Four Alternative Force Structure Emphases against Four 
Scenarios. 



These analysis challenges were extracted from Tom Allen’s QDR Workshop 
presentation during Wednesday’s plenary session. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Navy Analytical Challenges: 

- Littoral Challenge.. .Post Cold War Emphasis 

- Capabilities-Based Analysis for Resource Decisions 

» How to Map Support Missions to Warfighting Capabilities 

- Model Available to Define Capability to Meet Mission 
Requirements from Home Port to Crisis, However... 

» MOOTW Analysis Test Cases Varied and Changing 

♦ Need Stability of Dynamic Commitment Scenarios 

- Fleet battle experiment challenges 

» Data collection and preservation 
» Prioritization among real world operations, training, etc. 

» What arc the measures of success for FBE? 

♦ Innovation, translation to operations? 


Without the existence of a comparable Navy opponent for battles on the high- 
seas, the Navy will be faced with a “new” form of warfare... fights in the 
littorals. This new form of warfare will require new analytical tools to help 
define the appropriate force structures, doctrine, and tactics required to fight 
these crises. 

The IWAR/CPAM system is an end-to-end capabilities analysis method used to 
make resource decisions. The analytical challenge provided by this is to define 
the linkage between the support missions to their contribution to warfighting 
capabilities. 

The IWAR/CPAM process uses a model (FORSAT) that defines the capability 
to meet mission requirements from home port to the crisis area. The issue 
provided with this tool is the definition of the MOOTW scenarios. The 
scenarios are varied, changing, and growing by the day. The analysts, with 
current funding levels, cannot keep up with the development of the scenarios 
needed to describe the MOOTW conditions. 

The Fleet Battle Experimentation (FBE) program provides the Navy a means for 
evaluating new operational concepts within the fleet environment. Analytical 
challenges presented by the FBE is data collection and preservation, and thus 
translation into doctrine and operations. Due to the conduct of FBE within the 
active fleet, it necessarily takes a back seat to real world operations and other 
priorities during fleet operations. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Air force analytical challenges 

~ Expeditionary aerospace force focus 

» Vision, guidance, effects based operations with consideration 
of future warfare and risks guide modernization planning 

♦ Metrics to evaluate effects based operations 

♦ Detailed analysis needed to guide the transformation and 
innovation 

- Integration of analysis activities 

» Experimentation, wargames, battlelabs, ACTDs 



A previous CSAF decided that analysis did not fit well with his vision of the Air 
Force. He made the decision to decrease the size and organizational standing of 
the Air Force Studies and Analysis Agency (AFSAA). This decline has had 
many negative aspects, including the separation of blue suit analyst from 
contractor supported analysis. In many cases, the linkage between the analysis 
and decision making has been lost. At the same time that the analysis capability 
has been drawn down and de-emphasized, there has been an increase in analysis 
“911” calls from various sources such as CSAF, Congress, AF/XP, etc. 

The Expeditionary Aerospace Force concept offers many challenges. The 
combination of vision, guidance, and the consideration of effects based 
operations combine to guide modernization planning. During this planning 
process, future warfare and the decision risks involved must be considered. 
These radical changes in the AF doctrine and operations must be guided by 
thorough analysis activities that will guide the transformation and innovation. 

The need to systematically link all analysis activities is desperately needed to 
include the information and knowledge gained from the various activities, such 
as experimentation, wargaming, battlelab initiatives and ACTD’s /ATD’s 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Air Force analytical challenges (continued): 

- Model-centric focus limiting analysis toolkit 

» Scenario development and data collect time consuming, 

» Need “scoping models” to quickly assess alternatives. 

- Global vs. local optimization 

» OR community has the training to understand both. 

» Need tools to combine local (service specific) suggestions into 
global (Joint) solution. 

- Technology balance 

» Need to be careful of chasing technology at the expense of 
intellect. 

QB 


The current emphasis on M&S as the primary analysis tool, limits the capability 
of the analyst. Scenario development and data collection required for high- 
resolution models are extremely time-consuming tasks and thus limit the 
analyst’s ability to support quick-tumaround or small analysis questions. A gap- 
filler or “scoping model” is required to quickly assess alternatives that may 
subsequently be studied further with the high resolution or campaign models. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Air force analytical challenges (continued) 

- Joint collaborative analysis NOT joint collaborative decision making 

- Customer for all ages 

» Decision maker of the future may be true customer 
» Warfighter or decision maker as customer? 

~ Impact of logistics on successful operations 

» How to conduct cost/benefit analysis on logistics improvement 
investments 

- Logistics in wargames and staff training exercises 

» How to raise awareness of the importance of logistics 
“ Impact of IW on logistics 

- Civilian on the battlefield 

» Impact of civilian non-participation once battle begins 


The customer for our analysis capabilities may very well be the decision maker 
of the future, rather than current decision makers. Given this focus, how do you 
structure the analytical community’s activities to meet the needs of t his 
customer. 

The impact of logistics on the battlefield has not been adequately modeled in the 
current modeling tools. For example, modeling the impact of logistics on 
successful operations requires the need to do things such as cost/benefit analyses 
on logistics improvement investments, and quantifying the costs and benefits 
with respect to combat operations. 

In order to raise the awareness of the importance of logistics in senior leadership 
and planning efforts, there is a need to include logistics considerations in 
wargames and staff training exercises. Rather than allowing the decision maker 
to assume away all the logistics activities, they must be included in their 
thinking processes. 

What is the impact of information warfare on the conduct of logistics and thus 
the conduct of combat operations? For example what is the impact on combat 
operations when the logistics system supported them is attacked and brought 
down by information warfare operations? 

The impact of civilian participation on the battlefield needs to be considered and 
what the impact of their loss will be. For example, when the shooting began 
during Operation Desert Storm the UPS pilots providing logistical support 
refused to participate. 


WG3-19 




WG #3 New Analysis Challenges 

Session Summary 


♦ Marine Corps analytical challenges 

- Analytical Resources 

» Number of analyst driven by manning requirements NOT 
analytical requirements 

» Less analysts and more complex environment producing 
increased analyst workload 

- Identifying interrelationships between factors 

» JSF strikes vs. helicopter medevac during carrier operations 

- Increased expectations for operations analysis 

» More analytically savvy leaders expecting more and better 
products 

» Subordinate officers have unrealistic expectations 

♦ M&S is panacea 

♦ Every situation require.s a study 



The USMC is facing several analytical challenges: conflict of analytical 
resources and requirements, increased expectations of analysis; both leading to 
the need for a comprehensive analysis for the future. 

The number of USMC analysts is driven by manning requirements, not by the 
number of analytical requirements that the USMC is facing. This combination 
of more complex analytical problems and a small number of analyst combine to 
create a greater workload for each USMC analyst. 

The inability to define and analyze the interrelationships between systems, 
missions and functions provides an analytical challenge. For example, 
increases in JSF sortie generation rate allows the USMC to strike more targets 
ashore, but at the same time consumes deck space that may have been used for 
helicopters that could evacuate wounded from the ship. How are diverse, yet 
interrelated factors traded off in a comprehensive analysis? 

Increased expectations of the analytical community are spawned from savvy 
senior leaders who are demanding more and better analytical products. On the 
other hand, not so savvy mid-level leaders introduce unrealistic expectations for 
the community. They bring expectations that M&S is a panacea and that every 
question needs an analysis. 
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WG #3 New Analysis Challenges 

Session Summary 


♦ Must consider/use other analytical tools 

- Not just M&S 

♦ Analysis plan for the future 

- Focus on Operational Maneuver From The Sea (OMFTS) 

- Event vs. time driven analysis 

» Must incorporate lessons learned into analysis planning, not 
just blind adherence to early planning 



The analytical community MUST consider other tools rather than just focusing 
on M&S to answer every question. 

Analysis plan for the future focuses on Operational Maneuver From the Sea 
(OMFTS). Lessons learned must be incorporated in the planning and execution 
of future analytical activities. 
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MORS Workshop 

Evolving the Practice of Military 
Operations Analysis in DoD 

Wiprlpng Group #4 

New Analysis Tools arid Methods 

. .,........... 

; Chair: LTC Dan Maxwell 
Co-Chairs: John Furman and Dr. A1 Brandstein 

February 29 , 2000 through March 2, 2000 
Naval Postgraduate School, Monterey, CA 


The Charter of the New Analysis Tools Working Group was to identify 
emerging tools and techniques for conducting operations analysis, given the new 
challenges that face the military operations research community in the 21st 
century. 

To accomplish this, MORS leadership assembled representatives from domains 
that are not normally part of the traditional MORS population. Experts with 
expertise in C4I systems, intelligence, policy analysis, artificial intelligence and 
knowledge management were present. This eclectic set of skills resulted in 
information exchange that exceeded expectations — a vision focused on 
providing a foundation of consistent and compatible data, as well as an 
architecture that could provide connectivity between command and control 
systems and analytic modeling environments. 

The kinds of connectivity and consistency that are envisioned in this product are 
believed to be absolutely essential for providing commanders and staffs in the 
field with flexible automated planning support, and keeping analysis relevant in 
the highly uncertain future that the DoD faces. 

Perhaps most importantly, it was concluded that for the most part there is no 
technological impediment to achieving the vision that is articulated herein, and 
that these kinds of efforts are an essential part of achieving the knowledge 
management “culture” that has been implemented successfully in many 
commercial enterprises. 
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Working Group #4 Presentations 


♦ MAJ Curt Doescher, Modeling and Simulation Standards 
Development 

♦ Dr. David Homer, A Standards-Based Methodology for 
Throughput for Use in Movement Representation in Modeling and 
Simulation 

♦ Ms. Kay Burnett, Conceptual Modeling of Foreign Command 
Decision Processes 

♦ Dr. Mike Hieb, Data Alignment - C4I and M&S 

♦ Ms. Elaine Simmons, Employing Data Warehousing Technologies 
in Support of Analysis 

♦ CPT John McKitrick, Army Flow Model 
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WG #4 New Analysis Tools and Methods 
Focus Questions 

♦ What currently emerging developments in analytical approaches to 
complex systems can be applied to challenging military operations 
analysis problems? 

- Consider areas such as chaos, complexity theory, etc. 

♦ To what DoD analyses, particularly the new analytic challenges 
defined by WG3, can these approaches be applied? 

♦ What recommendations do you have to improve the existing 
analytic practice in the near term by using new analysis tools and 
methods? 

♦ What recommendations do you have for areas of research for the 
next generation military operations analysis? 



The MORS leadership asked the working group to identify developments in 
analytical approaches to complex systems that were emerging. While some of 
the areas were explored in the course of discussion, it is clear that there was 
much relevant research that was not able to be addressed in the compressed time 
schedule of a three day workshop. 

We attempted to map our discussion to the products of working group three. 
However, this topic is better addressed by the report of the synthesis group. 

The group did deliberate and prepare a set of recommendations on actions that 
could be taken by the DoD to improve the analytic practice, as well as improve 
the coordination with other domains of expertise. 
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Objective 


♦ To identify emerging techniques and tools that will support 
military operations research. 

♦ To map these concepts to the Department’s analytic challenges 

♦ To develop a set of near term investment recommendations for 
the MORS Sponsors’ consideration 

♦ To develop a set of longer-term research and development 
recommendations for the MORS Sponsors’ consideration 




Our objectives were derived directly from the focus questions that were 
presented to the group by the MORS leadership. 

One of the challenges for this working group was to identify real needs for 
improvements in tool and/or data capabilities given knowledge management’s 
emphasis on better exploiting existing technical capabilities. We searched for 
existing capabilities to exploit and attempted to organize our recommendations 
based on the vision you will see in a later slide. 
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Assumptions and Scope of Exploration 


♦ Knowledge management technologies will provide a 
“repository” in which these tools and techniques will be made 
available to analysts. (Limit WG scope to models, data and data 
development.) 

♦ Earlier objectives in the modeling and simulation master plan 
regarding reuse in modeling, simulation and analysis still hold. 

♦ The hardware and software infrastructure will support 
interoperability of systems (e.g. HLA and DIICOE). 

♦ Un-addressed deficiencies identified in the analysis functional 
working group report remain valid. 




Because the solution space was potentially vast and unmanageable in a 
workshop setting, the group made initial assumptions to bound the discussion. 

In some cases the assumptions were found to be to restrictive and were 
necessarily relaxed. 

The first assumption is that KM experts will create a repository infrastructure, 
and that infrastructure will be available for use to store and distribute analysis 
domain knowledge. 

Analysts and the DoD have been modeling and simulating for a long time. 

There is an existing set of recommendations in the DoD Modeling and 
Simulation Master Plan. These recommendations are assumed to be good, and 
future recommendations should deviate only when needed. 

There are known architectures available to us to move information between 
analytical models that could facilitate a knowledge management oriented system 
and culture. One example is the High Level Architecture (HLA). There are also 
known architectures to move information between near Real-Time C4I systems 
such as the Defense Information Infrastructure Common Operating 
Environment. While it is logical to assume that these architectures will ensure 
interoperability, discussion with experts from the C4I domain revealed that the 
assumption does not currently hold with respect interoperability between 
analytical models and C4I Systems (i.e., the HLA and the DII COE). 

The Analysis Functional Working Group has a valid list of analysis deficiencies 
that needs to also be considered. We anchored on how to do big studies better 
then started to look at minute-long decision cycles (from the analysis working 
group) to balance-out this discussion. 
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Assertions 


♦ Although warfare doctrine and military missions are evolving, 
there are constant/recurring analytic requirements that can and 
should be planned for: 

- Force assessment 

- Planning and execution 

» Deliberate planning 
» Crisis action planning 

- System effectiveness and trade-off analysis 

~ Concept and doctrine development and assessment 


EBB 


These missions are fairly constant in the workspace of military operations 
research. There is ever reason to believe they will continue to be the main 
missions of the future. They form the space around which to structure the new 
tools for analysis challenges. 

Because the group began deliberation at the same time WG 3 was meeting, a 
generic set of decisions (that could be supported) by analysis were used to 
stimulate discussion. Over the course of the workshop, the group considered 
WG 3 inputs as they arrived. 

One notable possibility that was added by the C4I domain was the existence of 
decision aids that are either currently embedded, or planned for in near real-time 
command and control systems. These systems would benefit from analytically 
sound underlying algorithms and the group felt strongly that the analysis 
community will need to be an active part of their development. 
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Assertions (Cont.) 


♦ Accomplishing these analytic missions require: 

- Descriptive tools (models and simulations) 

- Prescriptive tools (choice and resource allocation) 

- Reliable and consistent data 

♦ A significant body of v^ork is underway in the DoD that 
advances the state-of-the-art in modeling “infrastructure.” 

- Corrects deficiencies inside mission areas 

- Fails to capitalize on related work (schedule and resource 
constraints) 

- This results in inefficient and inconsistent modeling and analysis 
efforts. 




The taxonomy of models we see here include descriptive representations of the 
military world and also prescriptive models — those that prescribe a solution to 
the budget and capability trade space. We need efforts in both arenas to produce 
the analysis capability to serve the profession in the future. 

It was further asserted during the opening session that accomplishing these 
missions must be accomplished within shorter and shorter timelines. 
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Analysis Process in Context of Knowledge 





During the conference, we saw this knowledge management framework (the 
blue boxes) as a hierarchy of knowledge products built on the base of known 
facts and derived meanings. One of the insights from our work this week is that 
the application of knowledge management within the analysis community is 
really a process of becoming conscious of the importance of building and 
sharing the knowledge of our analysis enterprise — both subject area and 
analysis expertise. 

Knowledge products are produced by the analysis process (shown in yellow). 
The essence of analysis has always been to create knowledge from the raw 
materials of facts and meanings. 

One aspect of wisdom — the highest level knowledge product shown in the 
original framework (and missing from the far right side of this diagram) — 
results from continuing to develop and use analysis capabilities over time. 
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This is a slightly different view of the analysis process and products from the 
previous slide. It highlights the product areas that the tools working group 
might consider as areas for improvements. 

Much of our discussion centered on tools and data, particularly the need to 
integrate information generated from multiple tools, sequence data flows from 
one tool to another, and to ensure that the data objects were consistent such that 
they could be used as is or logically transformed to meet modeling 
requirements. The term “model,” defined in the broadest sense as the abstract 
representation of any problem, was frequently discussed as a key tool. 

Another emphasis area was to reduce the data-to-tool-to-information cycle time 
sufficiently to meet the needs of the customers with very short decision 
timelines. Traditionally, these customers have relied on back-of-the envelope 
estimates — aka “quick and dirty” analyses — to offer insights to support their 
decisions. Since these decisions are frequently high-level and wide reaching, 
the analysis community would like to design tool and data solutions that offer 
both a high quality and timely response. 

Decision support needs centered on visualization tools — the ability to clearly 
represent complex decision spaces, dissimilar tradeoffs, and the like — that 
have not yet been identified. The group identified KM practice as their 
preferred solution option to identify and reuse methodologies and measures. 
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What Should We Know? 


♦ Assets on hand 

- Authorization/on hand data 

♦ Investment strategy 

- Equipment aging.., succession analysis 

- Personnel development 

- Life cycle cost (estimates and historical data) 

♦ OPTEMPO 

“ Current missions 

- MTW readiness/training 
“ Spikes 

♦ Beliefs about the future 

- Future OPTEMPO 

- Personnel turnover 

- System effectiveness 



One of Vince Roske’s main points in his introductory remarks was that we 
“don’t know what we know.” OR has in recent years concentrated on the 
mechanical and physical aspects of predictive modeling. This has taken us 
away from the original roots of the science, which was grounded in the 
observation of combat processes and their improvement through study. 

We need to return to these roots and use KM to acquire the information we do 
know, and catalog it. Exercises, OPTEMPO, and force structure data must be 
treated as important information that is archived and made accessible to all 
analysts. 
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Key is to balance standardization with ability 
to exploit domain-generated expertise 


Analysts are called to serve a variety of customer, service, mission and 
supporting function domains. This slide shows a representative set of domains 
— there are others. The point is that each domain requires specific analysis 
perspectives and domain expertise. Analysis capabilities are developed within 
these domains to meet customer needs. KM offers a methodology and mindset 
to share information across domains that is represented here as the center 
“Metadata Integration Architecture.” 

“Metadata” are data that explain and establish the context for data within a 
conceptual system. One foundation of the analysis process is to generalize or 
abstract a problem to a level where it can be addressed with available time and 
resources and at a level that is consistent with a customer’s information needs. 
The elements of these abstractions can be considered metadata and have the 
potential to be reused as is or provide a good starting point for future 
assessments. 

The point is to exploit our body of knowledge to increase the responsiveness 
and quality of analyses by making best practices easily accessible. The term 
“standardization” is used as a shorthand for “data objects, algorithms, 
methodologies, measures and other analysis elements that are acknowledged 
best practices.” The balance is a delicate one — need for communication across 
domains with need for flexibility to respond to customer needs. 
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The Multi-Disciplinary Team of the Future 
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etc. 


Early OR practice flourished because analysts were able to assemble multi¬ 
disciplinary teams, and to bring structure to the problem solving process. Over 
time OR evolved into a discipline of its own, and reached out to other 
disciplines as needs were perceived. 

The challenges of the 21st century will require advanced, very complex 
technologies in all domains. This implies that no one person, or discipline, will 
possess the expertise to assemble a robust system, or even adequately describe a 
complex problem. 

An example of successful collaboration of this type is taking place in the 
JWARS development project. Descriptions of the JWARS software 
development process and project are available at http://www.dtic.mil/jwars. 
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The group developed an “architecture” of systems, software and models that 
was perceived to be necessary to meet the needs of military leaders, and defense 
managers in the twenty first century. The kinds of unprecedented connectivity 
and consistency that are envisioned are required to keep analysis and analysis 
products relevant inside the short decision cycles that are emerging. It is also 
necessary to achieve a culture of “knowledge management” in which the system 
of systems we are attempting to analyze has achieved a complexity that is 
beyond the comprehension of any one person or discipline. 

There are three primary nodes in this network of systems. First, there are near 
real-time command and control systems. These systems are used by operational 
planners and commanders that are tasked to execute the nation’s military 
missions. In the information age military, they exist in some form or another 
from the NMCC, down to the soldier in the foxhole. The second node is 
analysis models. These models, ranging from simple spreadsheets and decision 
tables to large theater simulations are used to develop and evaluate plans, 
choose courses of action, and manage the DoD. The final node envisioned 
consists of training simulations. Again, these simulations vary in scope and 
complexity. 

All of these nodes have a common requirement. They all need data (both 
primitive and meta) to function. If we are to achieve the envisioned 
interoperability it is absolutely essential that this data be logically consistent and 
portable across domains when desired. 
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One fact that emerged is that the C4I community is embedding decision aids 
into emerging command and control systems. These decision aids are 
effectively models that operational staff members and commanders will use to 
make decisions on those short timelines we see emerging. It is important that 
analysts (and others) be involved in the development of these algorithms to 
increase the likelihood that the underlying logic is sound and consistent. 
Moreover, this kind of involvement is necessary if the DoD is to achieve its 
objectives with regard to the use of software and achieve a more cost effective 
maintenance stream for models and data of all types. 

In theory the DII COE architecture provides vehicles for data communication 
from live systems to simulations. However, it was learned that the cuirent data 
representations in the C4I systems and analytic models will not support this 
objective. Moreover, there are similar inconsistencies in the data models used 
between the C4I and training domains. 
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An example illuminates how this system would improve our operational 
decision making. Envision a requirement for CINCPAC to deploy humanitarian 
assistance to East Timor. There is a need to provide logistical infrastructure to 
rebuild or secure this nation. Also, assume the planning process produced a 
contingency plan that includes identification of the forces that would be sent on 
this mission. 

The plan and all of its assumptions have been described in some analysis model 
(Perhaps in an OPLAN Format) and reside in data. At the time the requirement 
to execute is identified, the data in the plan are compared to ground truth data. 
The army database on current forces identifies the fact that the 25th Supply and 
Transport Battalion which was in the plan is not available because it is 
deployed; and the TRANSCOM data shows that the C-5 that are necessary to 
meet the timeline are committed to some other high priority mission. What has 
happened is we have used existing data to quickly illuminate assumptions that 
were valid at planning time, but are not at execution time. Operators are then 
quickly able to focus on managing the exceptions. 

Added to the previous vision are. data models for each of the three primary 
nodes. It is essential to realize that there are separate and distinct data models 
that must be aligned and synchronized to achieve interoperability. 

The deliberations in the group revealed that there are fundamentally no 
significant technological impediments to achieving this vision. The obstacles 
largely relate to achieving a sound common data model, and developing a 
“cultural will” to expose data to other domains. 
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WG #4 New Analysis Tools and Methods 

Emerging Developments in Analytical Approaches (1 of 6) 

♦ The Army standards category coordination process appears to have 
potential for improving the consistency of, and access to, modeling 
resources for analysis and reuse. 

- Air Force and Navy are developing similar approaches. 

- The standards structure provides a foundation for achieving 
knowledge management. 

~ Encodes analysis domain expertise. 

- Could migrate into a best practices compendium. 

- Standards need to be well documented (open peer review, clarity). 

- Process is as (or more) important than the resulting standard. 


The Issue is Good Meta-Data 



The Army presented their evolving standards category coordination process. It 
is ongoing and both the Navy and AF have undertaken efforts to develop similar 
programs. These efforts provide a foundation for KM rules and provides 
domain area expertise. It is an existing metamodel of analysis domain 
knowledge. Moreover, the structure provides an emerging link to the training 
and C4I domains. These efforts are emerging because there is a recognized 
need to achieve consistency across domains, if other efficiencies promised by 
technology are to be realized. 

The group also provided “seeds” to support further thought on this concept, so 
that one could possibly avoid some of the pejorative connotations that 
accompany the term “Standards.” 
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WG #4 New Analysis Tools and Methods 

Emerging Developments in Analytical Approaches (2 of 6) 


♦ Conceptual modeling and MSRR efforts appear to provide a good 
prototype for the content of a DoD M&S Imowledge management 
system with regards to military force behavior: 

” DMSO MSEA investments provide a foundation (direction) for 
interoperability/consistency. 

» C4I compatibility is a significant shortcoming (e.g. terrain). 

» Effort is in early stages. 

- There is a significant issue associated with opaque (undocumented) 
data generation and transformation. 

» “Life cycle” costs of models 

» Data stream 

- IWEDA and NRMM are examples of effective reuse by C4I and 
analysis. 


The conceptual modeling and MSRR efforts were identified as a method for 
encoding force behavior domain knowledge. Group members identified efforts 
around the community that are similar to the work that was presented. The 
group identified weaknesses in the technology but each implementation brings 
the community closer to describing force behavior in a manner that will support 
more efficient modeling, and consistent data exchange. 

Two examples of effective reuse across domains exist in JWARS. One is the 
Engineer Research and Development Center Vicksburg’s application of the 
NATO Reference Mobility Model in both C4I systems, and simulation. They 
map characteristics of trucks to trafficability networks. Use of this “model” is 
also envisioned in WARSIM, the land component of JSIMS. ARL’s Integrated 
Weather Effects Decision Aid (IWEDA) is being used in command and control 
as well as JWARS. This attempt at reuse identified shortcomings that are now 
identified research objectives of the Army Research Lab (ARL). 
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WG #4 New Analysis Tools and Methods 

Emerging Developments in Analytical Approaches (3 of 6) 


♦ There are technologies emerging to better “automate” data 
generation and management. 

- To wit: Army Flow Model and JDS Data Warehouse. 

- We lack a consistent understanding of “source” data and how to 
propagate it. 

- DoD Standard Data Dictionary is a start. 

- Inconsistency between C4I systems and models. 

- Hope exists (ERDC’s NRMM is “consistent”). 



The JDS data warehouse presentation shows technology that can achieve a 
“central” repository for data management and distribution. Web technologies 
could in fact be used to distribute this repository so that domain experts retained 
control over their data. 

The Army has a tool called the Flow Model that deconflicts their standard 
databases — e.g.. lets us know how many trucks are in the army today. This is 
the kind of information that should be more broadly available to support both 
planning and execution. 

The DoD data dictionary that exists in the C4ISR world is not consistent with 
the objects typically found in analytical models and simulations. 
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WG #4 New Analysis Tools and Methods 

Emerging Developments in Analytical Approaches (4 of 6) 

♦ There are some “logically sound” prescriptive approaches to 
supporting decision makers that are emerging or have had staying 
power. 

- Marine POM decision analysis. 

- CAA’s “Analysis in a Rucksack.” 

- Features of success are; 

» Simple, sound logic. 

» The ability to accommodate multiple stakeholders. 

» Transparency. 

» Visualization....consistency and clarity. 



The group talked a lot about the need for quick and dirty tools that support 
decision makers and ability to reduce knowledge to context of decision. 

Recommendations vice data. 

USMC decision analytic approach is 23 years old. The stuff exists that could be 
used or adapted for the toolkit. 

There are properties of these tools and analyses that cause them to endure. 
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WG #4 New Analysis Tools and Methods 

Emerging Developments in Analytical Approaches (5 of 6) 

♦ The application of “quick and dirty” and “presentation” tools is 
a weakness in the analytic practice. 

♦ Discipline and commercial tools can contribute. 

- Value focused thinking. 

» Fundamental objectives (hierarchy). 

» Means objectives (network). 

- Commercial “modeling” tools. 

» Logical decisions. 

» Analytical.... Etc. 

♦ Data issues are a recurring theme. 


The manner in which quick and dirty tools are applied is a weakness in the 
current military OR profession. The historical reliance on large scale combat 
simulations as the primary “validated” approach to conducting analysis lacks the 
flexibility to address the broad range of challenges the DoD currently faces. 

That said, there are examples of successful analytic projects that do not rely on 
the large-scale simulations for achieving insight. A common theme of these 
successful efforts is they are focused on the decision situations most relevant to 
the decision makers, they are transparent, and they are based on sound, 
consistent data. 
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WG #4 New Analysis Tools and Methods 

Emerging Developments in Analytical Approaches (6 of 6) 

♦ There are advanced modeling concepts that appear to have 
some potential for solving some of our “hard” modeling 
problems. 

- Agent based simulation. 

» Human decision making. 

» Soft factors. 

- Mathematical approaches to fusion (sensor and data). 



There are some hard problems out there that the group agreed would bear large 
fruit if we could tackle them and apply some quantitative science to them. 
Human decision making is not well represented today in analytical processes 
and models. Finally, there is some highly important Bayesian fusion work that 
holds the potential to develop sensor and multi-resolution theory. This work will 
also advance the ability of analysts to study and assess ISR activities. 
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WG#4 New Analysis Tools and Methods 

Potential Applications for Each New Approach 


♦ The identified approaches provide a foundation that can support 
multi-disciplinary teams in meeting new ANALYSIS 
challenges. 

- “Warm” consistent data set. 

- “Modular” set of models to draw on and build from. 

♦ There are holes (there always will be). 

- New approaches can be added to the base of knowledge. 

- Commercial tools can provide a “workbench.” 




Applications: ran some vignettes. Seemed to be able to map to all of them. 
Want over 50% solution on data, 80% solution on models. 

Will always be new analytic challenges. Need to build foundation. 
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WG #4 New Analysis Tools and Methods 

Near Term Recommendations (1 of 3) 

♦ Increase DoD emphasis on supporting inter-service collaboration 
toward standards. 

- This builds on the existing service efforts. 

- Possible organizational models are: 

» Existing JTCG (JMEM’s) is a possible model. 

» MSRR board of directors. 

- This should include Joint rules for reconciliation of standards. 

~ Establish a baseline (i.e. a minimal standard for meta-data). 

♦ Use limited experience as a foundation for KM implementation. 

- Formal plan for achieving the vision. 

» Senior leader support and involvement. 

» Deliberate investment strategy. 

» Uses the multi-disciplinary team. 

» There should be a co-evolution of culture and funding for 
technology. 


Work is starting; let’s get some resources into it. Needs joint rules to reconcile 
standards — not saying use same set but must be logically consistent. 

Need to build a formal plan not just throw money at the problem and touch all 
the communities. 

As plans become more complete and culture evolves, fund accordingly 
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WG #4 New Analysis Tools and Methods 

Near Term Recommendations (2 of 3) 


♦ The Analysis community should be represented in the DII COE 
process. 

♦ The DoD should develop and resource a strategy for collecting 
“other source” data, and applying KM. 

- NTC exercise data. 

- Historical deployment data. 

- Lessons learned data. 

- etc. 


ffii 


Decision aids being developed in C2. We’re either there or miss the 
opportunity. 

Lessons learned hits come from behind the iron curtain. 
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WG #4 New Analysis Tools and Methods 

Near Term Recommendations (3 of 3) 

♦ Encourage the creation and use of “quick and dirty” tools. This 
should be accompanied by: 

- Rules of the road for using data. 

- Best practices. 

- Mechanisms for: 

» Timely peer review. 

» Documenting use. 

» Allowing evolution. 






The working group shared a common vision that the community should let “a 
thousand flowers bloom” with respect to quick and dirty tools that lie outside 
the formal modeling constructs. These tools hold the potential to free us from 
the constrained environment of the large-scale simulation and return some speed 
and flexibility to our science. The problem with this philosophy, however, is to 
ensure accuracy and consistency in results while allowing flexibility in 
application. 

Rules of the road for using data would allow the sources of data to have a say in 
the use of their numbers, while a “Best Practices” guide would help to maintain 
some consistency in approach. Finally, the real value of the KM paradigm 
comes clear in providing a mechanism for peer review and documenting the 
usage of technique. This should speed re-use and allow for rapid evolution of 
technique. 
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WG #4 New Analysis Tools and Methods 

Recommendations for Future Research 

♦ A detailed review should be conducted that maps ongoing research 
to identified shortcomings: 

- Advanced concepts that “automate” human decision making 
(C2). 

- Intelligent Agents. 

- Inference/Fusion. 

- Visualization. 

- etc. 

♦ An investment strategy should then be developed to fill the 
identified holes. 




The DoD must invest in the research to allow us to quantify the human decision 
making process, use intelligent agents, and tackle the fusion problem so as to 
better represent intelligence processes in analysis. Visualization techniques, 
both to increase human absorption of information and to yield better displayed 
information for decision makers, must also be investigated with energy. 

The Department must conduct a comprehensive review that maps research into 
identified shortfalls. We don’t know what we don’t know, about the basic 
research being conducted in these areas. Before we create an investment 
strategy, we need to get the holes right and figure out how to fill them. 
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The future framework will require a warm bed of consistent data and models 
that will operate inside the existing decision cycles. The vision is to achieve a 
fairly seamless linkage between real systems and the families of training and 
analytical models and simulations. 

Each component has its own data (represented in purple) that are particular to 
its domain and independent of the other areas. This data is highly relevant and 
important, and must be available to all through the KM paradigm. 

Analysis methods are important, but the key to turning around analysis more 
quickly is data. A warm bed of data, COTS tools, plus a robust training program 
will help us utilize what we’ve got. Imbedded tools in C4I systems will 
generate displays — need to help design these. 

Changing the tools will not speed the analysis process alone — we need to 
change the foundation that we’re working with to give more speed. KM is key 
here. CINCS have existing contingency plans, and analysts often find the 
closest, most relevant plan to build from. We need a richer set of things to work 
with, and netted real world and analytical systems in a KM paradigm will give 
us this. 

Over time, models will generate information that becomes another data source. 
Models would generate new metadata that should be included in this picture. 
Analysis cases we build can live on the shelf ready for reuse. 
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This presentation provides the comments and insights of the Synthesis Group 
for the “Evolving the Practice of Military Operations Analysis in DoD” 
workshop as presented during the final session. The Synthesis Group attempted 
to pull together common themes from all the working groups, as well as to 
provide insights about how separate themes developed by these groups might be 
harmonized to support a common vision and meet the overall objectives of the 
workshop. 
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Synthesis Team 


♦ Dr. Stu Starr, FS (Chair Synthesis Emeritus) - MITRE 

♦ Dr. Jerry Kotchka - Co-chair - LM/NESS 

♦ Dr. Bob Hinkle (WG-1) - DUSA(OR) 

♦ Dr. Tom Allen (WG-2) - IDA 

♦ Dr. Mike Sovereign (WG-3) - IJWA/NPS 

♦ Dr. Mark Youngren (WG-4) - MITRE 

♦ Dr. Roy Rice (Executive Council Rep) - TBE 



Members of the team are listed on this slide. Dr Stu Starr, FS, Working Group 
Chair, was called away the day before the workshop opened and was unable to 
attend in person. Despite that, his massive research and coordination with the 
other working group chairs leading up to the workshop enabled him to gather a 
great deal of background material, as well as to provide an initial shape to this 
outbrief. Without his participation and virtual leadership, the final product of 
the group would have suffered immensely. Dr Jerry Kotchka stepped in to ably 
coordinate the activity in Monterey. The team included members from each 
working group and met daily with the chairs and cochairs of those groups, both 
to highlight common areas and to request effort on specific topics that would 
help the Synthesis team in their efforts to create a follow-on action plan to help 
the military operations research community achieve the overall vision of the 
conference. 


Synthesis WG - 2 



Vision 


Since analysis is both a product and a process, 
our vision is a reemphasized and revitalized practice of analysis 
to more effectively and more efficiently 
connect the decision maker and analyst 
so that 

the final product is not the primary result. 



Toward this end, the Synthesis Group asked each Working Group (WG) to 
articulate a vision that would capture the end state resulting from full 
implementation of their group’s reco mm endations. Based on their inputs, the 
Synthesis Group articulated the vision shown here. We believe this vision 
transcends and incorporates the WG visions and describes how emerging 
capabilities such as collaboration and knowledge management can transform the 
practice of military analysis. The emergence of powerful communication and 
computer-based capabilities is already changing the economic world. Where 
before the focus was on production, the emerging giants of e-commerce are now 
succeeding through their acquisition and management of information. The most 
successful and most profitable of these businesses are those that enable the 
highest level of information sharing, so that opportunities are quickly identified, 
decisions made based on the full amount of information and insight available in 
the work force, and then action initiated appropriately. The knowledge resident 
in every worker is rapidly connected to the decision makers so that timelines are 
reduced and customer needs satisfied, optimizing company effort and taking full 
advantage of the knowledge base of the organization. In a like manner, military 
analysis in the future will need to be fully integrated, with analysts and decision 
makers so closely linked through technical and personal means that the 
maximum benefit will flow to the organization. In fact, the interaction will be 
such that the final results of any study will surprise no one, since decision 
makers and analysts will both be engaged throughout the analytic process, 
allowing the strengths and insights of each to be fully incorporated. Such a 
construct will revitalize the practice of analysis, take advantage of the emerging 
capabilities, and better address new challenges confronting analysis in the new 


millennium. 
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This slide illustrates one view of the current practice of analysis. It shows that 
analysis is focused on one goal: helping decision makers choose among 
alternatives. The four elements analysts support in this activity include: 1) 
helping the decision maker define the issue or questions; 2) helping to define the 
decision space so that a full range of alternatives can be considered; 3) 
establishing meaningful measures of effectiveness and criteria that will help 
show the strengths and weaknesses of the various alternatives; and, 4) then 
creating understandable, traceable and repeatable methodologies that will assist 
in this selection process. To do this, an analytic foundation must be in place. 
This foundation includes the people who contribute to the process, primarily 
analysts (but including a broad range of other scientists, engineers and subject 
matter experts); models, or the concepts, mathematical abstractions or other 
constructs that are used to decompose a problem and focus on key aspects; and 
the data, scenarios, assumptions and other factors that are necessary to support 
the process. The issue addressed by this workshop is how this framework is and 
will be impacted by the changes in the National Security and Technological 
Landscapes and how the practice of analysis should evolve to handle the 
challenges and take advantage of the opportunities provided by these changes. 
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The Analyst’s Dilemma 



“Whatever happened to elegant solutions?” 



Of course, the very complexities presented by the rapidly changing landscapes 
associated with this environment may have removed the ability for analysts and 
others to focus on the big picture. The new environment as assessed by 
Working Group 3 presents a conundrum; we are asked to provide insight on the 
complex interrelationships between factors input to and within our models (that 
reflect interrelationships in the real world), but we are also asked to reduce the 
complexity and increase the transparency of our tools and analysis. Although it 
may not be possible or necessary to reduce the complexity of our high-end 
models (recognizing of course that not all analysis requires large and complex 
models), open sharing of model data, assumptions, algorithms, etc., all within a 
set of agreed-upon standards and implemented through knowledge management, 
may increase the transparency of the models and analysis process among trained 
analysts. The analyst will always have to convert complex mathematical and 
structured logic into forms understandable to a decision maker without an OR 
background in all but the simplest cases. Thus, an elegant solution may not be a 
reduction in complexity to a handful of variables, but instead be a transparency 
and sharing among analysts that is communicated to the decision maker 
throughout the analysis (rather than at the end) in a clean and direct (if not 
elegant) manner. 
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Definition 


Dr. Ruth David’s definition of “Knowledge 
Management” 

- Process through which an enterprise uses its colleetive 
intelligence to achieve strategic objectives: 

» Strategy 
» People 
» Process 
» Technology 


Bi 


At the same time, there are potential solutions available, at least to address part 
of the changing environment, and to help our analytic community and other 
parts of DoD enterprise to adapt and prevail in the emerging environment of the 
21st century. Dr. Ruth David, our keynote speaker, defined knowledge 
management as shown on this slide. Inherent in her definition was the 
understanding that the enterprise using knowledge management was a “lean, 
mean, fighting maehine team” foeused on eommon objectives and with a shared 
understanding of the necessity and value of eaeh other’s contribution. Such a 
definition of enterprise may not reflect the current state of DoD, where often 
mutual contributions are not recognized and parochialism can break down 
teamwork. However, to the extent that shared goals and team work reflect DoD, 
and in the areas that knowledge management can help the department move 
toward this enterprise vision, knowledge management and its implementation 
surely should be embraced by the department. 
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A Framework for Formulating 
KM, Collaboration Recommendations 



The accompanying figure provides a business process re-engineering 
perspective of collaboration and knowledge management. The Synthesis Group 
concluded that if we are to successfully implement these capabilities in a 
dynamic environment, we must consistently address all of these factors. For 
example, attempting to field a technology that makes available all information 
to all members in a culture where classification, close hold consideration of 
alternatives, and fear that knowledge of a programmatic weakness could result 
in elimination of a system even if other elements are robust and essential to a 
military capability, can only result in failure. Before the technology can be 
designed, the culture must be addressed and pragmatic steps taken to show how 
such information sharing can actually result in overall benefits to the 
participating organizations and people. Concrete trust building steps can be 
followed by bigger steps ultimately implementing the full capability. Just as 
culture must be addressed, so must the inherent goals and objectives of the 
involved organizations and people. Many experts at this workshop agreed that 
the technological solutions for most of the knowledge management and 
collaboration techniques are already in hand. The biggest challenges will be in 
addressing the needs of culture, organization and people. 
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Returning to our analytic construct, we can see that changes in the national 
security landscape are already shaking some of the analytic pillars. Evolving 
from attrition-based warfare to effects-based warfare, understanding the 
implications of C4ISR systems and capabilities, and the emergence of a broad 
new array of asymmetric threats are all impacting the ability of the analytic 
community to develop reasonable measures of effectiveness as well as to 
develop clear and repeatable methodologies to assess these aspects of military 
activity. While these areas are outlined with the dashed lines (which appear in 
red in color versions of the chart), the technological landscape and emergence of 
new capabilities like KM are impacting the rest of the words to potentially 
transform the way we conduct analysis. 
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Challenges 

mtional Security Landscape* 

~ effects (C4ISR follow-on meeting) 

- Environmental - Threat (world is changing - are we?) 

- Ash Carter and Bill Perry (A/B/C Lists from QDR Workshop) 

- Technology insertion 

- QDR 2001 (strategy - ???) 

♦ Technology landscape 

“ Knowledge management 

- Collaborative analysis management 

- Commercial initiatives 




This chart provides more detail on the changes, with the circled “National 
Security Landscape” reflecting the circled area on the last chart. We’ve already 
hinted at the changing paradigm of massed armies meeting on a battlefield to 
the massing of effects at the appropriate time and place as the way current US 
military leadership is starting to think about warfare. In fact, to help think 
through the implications of such changes, MORS has conducted a number of 
workshops and will hold a C4ISR Follow-on Workshop in the fall to continue to 
evolve our understanding in this arena. Some of our speakers in this workshop 
have referred to the Ash Carter/Bill Perry work on who the threats of the future 
are. They range from an A List of direct threats to national survival which 
would include homeland attack by catastrophic terror agents, to a B List of 
threats to national interests that the current strategy seems to address, to a C List 
of important problems that do not threaten US vital interests (e.g. Kosovo, 
Somalia). In addition, new technologies require new analytic capabilities to 
measure the effects and potential contributions from such areas as directed 
energy, information technologies, unmanned systems and space. The QDR 
process may change the national military strategy, so the analytic profession 
will need to be responsive both in helping to determine the implications of 
strategy shifts as well as in implementing any new strategy in the analyses of the 
future. 

Just as the security landscape is changing, so is the technology environment in 
which analysis is conducted. This workshop has highlighted KM and 
collaborative capabilities, but there are a number of other initiatives, such as 
simulation based acquisition and other e-commerce activities that the analytic 
community needs to track and adopt where it makes sense to do so. 
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This slide shows an illustration of an ideal end-state for the current efforts to 
improve analytical practices. There are three families of models and 
simulations: M&S embedded in real-time command and control systems; 
training and exercise support simulations; and models and simulations used to 
support analysis (to include T&E analysis and analysis of non-warfighting 
issues such as infrastructure and business process improvement). Within each 
domain are a set of models and simulations tailored to the purpose at hand. No 
single model will apply universally across all domains, but in this ideal end state 
we have data, software objects, and process representations shared across the 
domains where it is logically consistent with the model and purpose. 

Infrastructure programs currently under development are assumed to be in place 
in this end state, to include the High Level Architecture (HLA) for simulations 
and the Defense Information Infrastructure Common Operating Environment 
(DII COE) linking all areas together. HLA federations which may include 
simulations widely distributed over space will exist where appropriate, but these 
federations should be standardized and validated for consistency across 
federates before use. The creation of “on the fly” federations that grab objects 
and federates scattered across the DII to create unique environments is probably 
still not possible in this end state because there is no guarantee that the 
representations will be internally consistent (consistency is not assured even if 
data is passed through an HLA), and it is unlikely that a single analyst or team 
located at a single site will be familiar with all of the objects or federates 
potentially available. Instead, these simulations may be combined through the 
use of collaborative analysis rather than software linkages. 
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All of the tools are presumed to be in a bed of warm data. That is, there is a data 
infrastructure that makes data readily available (with exhaustive metadata explaining the 
derivation and accredited use of the data) for all of these systems. For example, an 
analysis of real world capability should be able to draw on the same data set showing 
current status and deployments of units that a real-time command and control system 
might use. Similarly, an analysis of a future capability (for example, a 2010 force in a 
particular scenario) should be able to draw on a validated and commonly used set of data 
to represent the scenario, system performance, OPFOR performance, etc. Generation and 
storage of this data requires a significant resource commitment within the Department of 
Defense, but it is necessary to achieve the desired end state. 

A key feature of this environment is that the output of the models, as well as the results of 
specific analyses and the lessons learned from training and exercises, are also added to the 
common data base. This provides a useful source of information that can be of particular 
help to quick turn-around analyses, which may synthesize the outputs of previous models, 
exercises and analyses. 
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Tools and Techniques/Challenge Matrix 
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Part of the improved environmental end state is the development of new tools, 
techniques and methodologies to address the specific challenges of the future. 
Some of the new or emerging tools and techniques identified by Working Group 
(WG) 4 are shown on this chart to illustrate which show promise to address, at 
least in part, the challenges identified by Working Group 3. Those challenges 
fell into 6 broad categories: 1) changing characteristics of warfare in the 21st 
Century (New Wars); 2) inclusion of the effects of humans and human factors 
(e.g., morale, training) into models (Behavioral); 3) showing the effect of 
logistics and other support activities on warfighting outcomes (Support); 4) 
identification of complex interrelationships between input variables and 
between internal variables (processes) within models (Interactions); 5) explicit 
quantification of risk and uncertainty in analysis and in communications with 
decision makers (Risk); and, 6) the challenge of developing and gaining 
acceptance of analysis techniques that are not dependent upon traditional 
simulation or other models (e.g., linear programs) (Non Model). 

WG 4 identified many recommendations regarding new or emerging tools and 
techniques that should be explored and implemented where appropriate. These 
tools and techniques fell into 10 categories as shown above. The first category 
listed involves standards. There are some relatively new efforts underway to 
develop common standards that can be applied to the models and data used to 
support analysis; the Army Standards Category Coordination process is one 
example of this. DMSO sponsored efforts to develop common conceptual 
models and M&S Resource Repositories (MSRR) is another example. The 
group recommended that the fundamental algorithms underpinning many major 
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simulations in use be examined in open (peer review) forums and reconciled. Although 
a commonly accepted set of algorithms would be desirable, it is not necessary that all 
services or other users agree to a single representation of a warfare phenomena. We can 
agree to disagree, provided the differences are clearly documented, understood and 
made available to the analysis community. 

The second technique identified by WG 4 is automated data generation. The biggest 
single complaint from the working group was the lack of needed data and the difficulty 
and cost of generating or obtaining data even when it is available. Automated means of 
generating commonly used data would increase the responsiveness and reduce the cost 
of getting the data. 

The “Analyst’s Rucksack” is an example of using quick and simple tools and decision 
aids to fill in for large model deficiencies and directly support warfighters. The 
Workshop endorsed the development of tool sets of this nature. 

The commercial world is also producing an array of tools that have great utility for 
operations research analysts. Tools which have been around for several years, but are 
becoming increasingly sophisticated and easy to use, include statistical and data 
analysis packages, linear and nonlinear programming, process simulation and decision 
analysis software. Emerging products include neural networks, Bayes nets, colored 
Petri nets and other software that propagate inferences or influences. 

There is recent research that is producing some new techniques that may be applied to 
fix, in part, some of the areas that lack tools for analysis. Agent-based simulation may 
help in the analysis of doctrine, procedures and risk sensitivity, and recent 
mathematical models have been developed to assist in the simulation (and real world 
execution) of fusion between multiple sensors and sensor disciplines. This is not an 
exhaustive list, but are two areas that show promise. 

Performing analysis in the 21st century will likely lead to the resurgence of the multi¬ 
disciplinary team. However, this team will have a different composition than the teams 
in the early days of military OR, which were largely formed from hard sciences. 
Although some problems may still require physicists, chemists, etc., most work will 
require groups pulled from 3 broad areas. The first consists of domain experts, which 
will often include warfighters, doctrinal experts, and other expertise in military 
operations. The second group are software and computing experts, as models and tools 
become increasingly complex from a software perspective. The third group, which 
helps to meld the other two together, consists of operations research analysts and other 
experts from related analysis domains, such as mathematics and statistics. We should 
no longer expect that a single military OR analyst be able to read and write code, 
develop mathematical models and possess a broad military background to include joint 
operations. 

The implementation of knowledge management techniques identified by WG 1 and 
presented by plenary session and lunchtime speakers will also help address the 
challenges to the practice of OR. These concepts are developed in detail in other 
sections of this report. 
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Modeling and Simulation Executive Agents (MSEA) have, in some areas, greatly 
advanced the state of the art in M&S. The environmental MSEAs have had a great degree 
of success and can serve as exemplars of the process that leads to significant 
improvements. They have developed data standards applicable to a wide variety of 
simulations (to include both training and analysis simulations) and real world command 
and control systems, and populated these data structures with data that is easily obtained 
through user-friendly business processes. They have also developed algorithms to 
transform the data into information useful to aggregated models; for example, the Army 
has developed techniques for automatically generating road networks from detailed 
NATO-standard mobility networks. 

One area that can be addressed through good knowledge management practices involves 
“other source” information. This data includes NTC exercise data, historical deployment 
data and “lessons learned” data. These data sometimes exist but are not necessarily 
recorded for posterity (e.g., much NTC information is outbriefed to the unit commander 
and then deliberately destroyed), not captured at all (e.g., historical deployment data), or 
are captured only qualitatively rather than quantitatively (e.g., lessons learned). These 
data, if resources are applied for capture and storage, can significantly improve our ability 
to validate models against “real world*” experiences. 

Finally, the group recommended that additional research be made in the area of 
automated decision making. Large simulation models require some automated 
representation of a human decision maker (e.g., a commander and staff in an analysis 
model or an OPFOR or subordinate commander in a training model) to provide 
reproducibility or reducing exercise/training costs. However, the current implementations 
are recognized to be insufficient and unrealistic. Automated decision making can also be 
embedded in real time decision aids, not to replace a human decision maker but to advise 
him or her. 


* We recognize that exercise data is not real combat data, but it is a useful surrogate that may be more 
accurate than outputs from abstract model and simulations. 
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Endorsement 


KNOWLEDGE MANAGEMENT 
HAS THE POTENTIAL TO ENHANCE 
OPERATIONS ANALYSIS 

Distinguishes “what we know, what we don’t know” 
Enhances Information Sharing and Collaboration 
Addresses the “Knowledge is Power” Paradigm 




While concepts associated with specific challenges were shown on the last slide, the 
Synthesis Group also felt that knowledge management can offer a more general solution 
to the broad issues confronting analysis as well. Some aspects of KM can already be 
endorsed in ways that do not threaten existing culture. In particular, KM has the ability to 
help distinguish between what we know and what we don’t know by helping the 
enterprise quickly tap the collective knowledge of individuals in the organization. This 
capability can provide an individual assigned to work a problem the capacity to quickly 
poll all experts in or out of the organization to assess what is already known or what is in 
the process of being studied about the problem. In fact, a properly implemented KM 
system will save enormous time and resources by helping the analyst and others rapidly 
determine what is already known as the basic starting point of any study. The same 
system, by enabling rapid connectivity between analytic practitioners, will also enhance 
information sharing and collaboration. When a study team, already focused on a specific 
area, realizes that the value of its research can be increased by adding just a small amount 
of effort in order to address a broader range of questions to meet the needs of a broader 
customer base, one promise of KM will be realized. And as the limited resources 
available in an organization self-organize to focus on the areas they do best in order to 
provide a broader base of insight and information to the organization, the people involved 
will clearly understand that their value-added is not in hoarding information, but rather in 
developing and sharing it in ways that enhance the overall capabilities of the enterprise. 
Value added will not come from just having information, since in the future information 
will be increasingly available to everyone; increased value added will instead come from 
the ability of individuals and organizations to share available information creatively to 
enhance the capabilities of the organization. 


Synthesis WG -15 





Action Plan 


Near 
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Far 
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DoD " Working Group recommendations are included as a backup to this briefing 
and provide a phased approach to KM and collaboration implementation by 
focusing on people, process, technology and product, as well as list 
challenges and promising tools and methodologies to meet those challenges 



The Synthesis Group identified a number of actions, some of which are shown 
here, that should become part of a long range action plan that the community 
should consider and flesh out over the next few years as it evolves the practice 
of military analysis in DoD. In particular, there are several steps that MORS 
itself can implement immediately that will have enormous potential to improve 
the ability of this community to connect with each other and better meet the 
over all needs of its membership. In addition to the items shown here, the 
working groups provided a number of general and specific steps that will help 
the community learn to harness KM and collaboration as well as to address 
emerging issues in the changing security environment. Generalized 
recommendations endorsed by the Workshop include working the people, 
process and technology problems in order to ensure a better final product. 
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Summary Recommendations 


♦ Vision/policy: develop and promulgate enterprise Vision, strategic 
directions. 

♦ Culture: take action to remove deeply rooted cultural barriers (e.g., dispel 
the myth that “knowledge is power”). 

♦ Organization: Ensure top management buy-in. 

♦ People: provide incentives to analysts to collaborate and share knowledge. 

♦ Processes: institutionalize processes to enhance collaboration and the 
sharing of tacit, explicit knowledge among analysts. 

♦ Analytic Tools: 

- Develop and emphasize joint standards in algorithm and data development. 

- Take action for developing and maintaining data warehouses. 

- Focus basic research and tool development to meet emerging pressures and 
challenges. 

QSS 


This chart summarizes the focus and potential impact of the specific actions 
recommended. First, cultural changes are a necessary prerequisite to effective 
transformation of the analysis community (e.g., the need to dispel the myth that 
“knowledge is power”). Second are several organizational issues that must be 
addressed. One of these is to ensure top management buys in to this 
transformation. Another includes reassessing how security issues can be 
addressed in an information centric, open environment. In the third area, 
decision makers must recognize that the analysts involved in all phases of the 
assessment process constitute a critical intellectual resource. They must be 
given incentives to stimulate them to contribute and draw from future 
knowledge warehouses. Tying all this together are the links between the vision 
and the high level guidance to the analysis process and the primary products of 
that process (i.e., effective, timely, rigorous analyses). To implement this 
linkage, a set of enhanced processes is needed (e.g., institutionalized processes 
to diffuse tacit and explicit knowledge) supported by a broad set of tools (e.g., 
knowledge warehouses, collaboration tools). These processes and tools are 
constrained by available resources (e.g., funds, manpower) such that a special 
funding mechanism may be required to satisfy time urgent joint needs. In 
particular, basic research must be continued in those areas where understanding 
of phenomenology is not available (human decision processes, lO/IW, etc) as 
well as the continuing development of analysis tools to address the emerging 
challenges. 
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Our Ultimate Goal 
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The Synthesis Group feels that by addressing the needs of the community to 
implement our vision, the practice of analysis can evolve from its current 
position of a limited toolkit and methodologies to address a limited number of 
issues to a complete tool set connecting analysts and decision makers to address 
the full array of national security challenges. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 1 of 7) 

♦ The DoD Sponsor implement knowledge management techniques, 
particularly to establish an intranet portal, in support of an 
important study with suitable scope, schedule, and content that 
partners DoD components Military Operations Research Society, 
industry and academia. The elements that should be considered 
along with strategy already discussed include: 

- People 

- Process 

- Technology 



This is our overarching recommendation, which detailed in the next series of 
slides under the headings of: 

Strategy 

People 

Process 

Technology 

Specific organizations that should be included are the Services, OASD(C3I), 
DTIC, MSIAC, DMDC, DMSO. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 2 oil) 


♦ People 

~ DoD and MORS lead an assessment of the capability to provide a 
baseline inventory of analysts for DoD operations research studies. 

- MORS assess its capability to establish a knowledge management 
professional development program for the DoD analytic enterprise 
that would include the identification of skills and products, the 
refinement of skill levels, and the training of analysts. 

*“ The DoD Sponsor evaluate successful skill-inventory systems (e.g. 
Northrop-Grumman and Y2K) for implementation within the DoD 
Analytic Enterprise. 


GS§ 


Industry experience and DoD case studies show that people is one of the most 
critical success factors associated with KM. The recommendations cited here 
are designed to provide the fundamental foundation for organizing and 
educating people using KM techniques being applied by industry. Key DoD 
organizations that could be involved in pursing this recommendation include 
ASD (C3I), DUSD(P&R), OSD(PA&E) and DMDC. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 3 of 7) 


♦ Process: 


- J-8 accept leadership with OASD(C3I) of an evolving analysis 
process for the purpose of improving analysis in a future analysis 
state. Notionally, the evolving process could be closely interfaced 
with an evolving knowledge management process as shown on the 

Skilled 
Analysts 



Instrumentation 
Plan for Data 
Collection 
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The working group recognized that the future (to-be) process incorporating KM 
is still to be defined. Nevertheless, this slide helps illustrate the important 
interrelationships related to collaboration, creation, and reuse of a well 
constructed analytical KM environment and extensible knowledge base. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 5 of 7) 


♦ Process continued: 

- The DoD determine the business rules for contributing analytic 
knowledge. For example, what are the current rules that control 
the access to past studies and how could these rules be changed to 
improve access such as those stored at Federally Funded Research 
and Development Centers. A special case is the identification of 
the business rules for administering and controlling an analyst’s 
portal. 

- The DoD Sponsor, with the assistance of MORS, identify new 
types of information required for future analyses (e.g. foreign 
demographics, information warfare related, economic information, 
national infrastructure related, chemical/biological, etc.) 

- The DoD Sponsor identify current Centers of Expertise required 
for improved analyses. 


IflBia 


The working group concluded that the first bullet was probably the most critical 
follow-on task for DoD to tackle. If this is not done, KM will fail. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 6 of 7) 


♦ Process continued: 

“* Take steps to systematize the organization of information by 
transposing principles from traditional library and information 
sciences to the electronic realm. 

~ Develop and use an Information Architecture that structures the 
design of information organization, labeling, navigation and 
indexing systems to support both browsing and searching. 

- Identify and use a set of metrics for the business case preparation 
and performance evaluation of the analyst’s portal. 

- Create knowledge-sharing incentives such as: 

» Knowledge-sharing inclusive performance reviews 
» Embedded authorial recognition 


The importance of performance reviews and embedded are mentioned 
repeatedly in the literature. The study done by the Working Council for the 
Chief Information Officers strongly supports this point. This study also stressed 
the importance of having an information architecture for effective design and 
administration of browser-enabled knowledge-based repositories. 
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WG #1 Management of Analytic Knowledge 

Recommendations (Page 7 of 7) 


♦ Technology 

- The DoD concentrate on applying COTS tools as a key critical 
success factor for building analytical KM portal. Example 
technologies include: 

» Thesaural browsers. 

» Personalized intranet portal. 

» Knowledge-sharing inclusive performance reviews. 

» Federated knowledge management. 

» Information-expiration mechanisms. 


lUllURY Or£RAllC«S (OCIETY J 
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WG #2 Enabling Collaboration 

Recommendations 

♦ People 

“ Find ways to identify (and evaluate?) Subject Matter Experts 
(SMEs). 

- Find a means to develop operational experience in Civil Service 
1515s and uniformed analysts w/o warfighting specialties. 

♦ Process 

- There is a spectrum of Collaborative analysis processes, and each has 
its strengths and weaknesses (bold shifts versus consensus). 

» Can organize for competitive analyses. 

» Can organize to search for a “common ground” solution. 

- Collaborative Analysis is a “team sport” and requires practice. 

» OA Support to Commanders in the field most effective when 
“Train as you Fight” has been implemented. 



Our recommendations have been collected under the headings of “People,” 
“Process” and “Product.” 
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WG #2 Enabling Collaboration 

Recommendations 

♦ Process (continued) 

- Reestablish fornial liaison/exchanges to improve the quality of 
collaborative analysis 

- Support independent, accredited sources for data to speed 
collaborative effort. 

- Find ways to identify data to source new applications and phenomena. 

- Data collection must be deliberate and done in such a way that it does 
not require additional effort by operators. 

- Synchronize data production with data demands and institutionalize 
production and update of long-lead items. 

» E.g., extend JMEM to include developmental systems. 

- Harmonize schedule of major collaborative projects and Service workload. 
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WG #2 Enabling Collaboration 

Recommendations 


♦ Product 

- Need improvements in literature search capabilities, identifying 
work in progress that may affect the project at hand. 

- Study documentation techniques must also improve, allowing for 
reproducibility of results and reuse of data in addition to 
description of procedure and findings. 
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WG #3 New Analysis Challenges 

Recommendations 


♦ Find methods to deal with. 

- New forms of operations. 

- Traceability of known factors and interactions. 

- Quick turn-around studies. 

♦ Nurture analysts and analytical thinking rather than focusing on 
model development. 

- Project more balanced view of model capabilities and limitations. 

- Reverse trend to big models as solution to all problems. 

♦ Characterize and communicate impact of uncertainties (e.g., 
Risk, behavioral factors). 




In summary, as the analytical community evolves it needs to find tools to deal 
with new forms of operations, find methods to trace known factors and 
interactions to their logical combat output and find tools that will provide 
accurate and thorough analysis in a short timeline. 

The DoD analytical community needs to nurture analysts and emphasize 
analytical thinking rather than focusing on simulation and modeling 
development. The community must project a more balanced view of model 
capabilities as well as limitations, and reverse the recent trend toward blind faith 
toward large models as the solution to all analytical problems. 

The analytical community needs to find means for characterizing and 
communicating the impacts of uncertainty. For example risk, such as decision 
risk based on the fact that various factors may not have been considered in the 
current analysis, need to be characterized and communicated to the decision 
maker,the analytical product customer. 
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WG #4 New Analysis Tools and Methods 

Near Term Recommendations (1 of 3) 


♦ 


♦ 



Increase DoD emphasis on supporting inter-service collaboration 
toward standards. 

“ This builds on the existing service efforts. 

- Possible Organizational Models Are: 

» Existing JTCG (JMEM’s) is a possible model. 

» MSRR Board of Directors. 

“ This should include Joint rules for reconciliation of standards. 

- Establish a Baseline (i.e. a minimal standard for meta-data). 

Use limited experience as a foundation for KM implementation. 

- Formal plan for achieving the vision. 

» Senior Leader Support and involvement. 

» Deliberate Investment Strategy 
» Uses the Multi-disciplinary team. 

» There should be a co-evolution of culture and funding for 
technology. 
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WG #4 New Analysis Tools and Methods 

Near Term Recommendations (2 of 3) 


♦ The Analysis community should be represented in the DII COE 
process. 

♦ The DoD should develop and resource a strategy for collecting 
“other source” data and applying KM. 

- NTC exercise data. 

- Historical deployment data. 

- Lessons learned data. 

- etc. 
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WG #4 New Analysis Tools and Methods 

Near Term Recommendations (3 of 3) 


♦ Encourage the creation and use of “quick and dirty” tools. This 
should be accompanied by: 

- Rules of the road for using data. . 

- Best Practices. A / 


- Mechanisms for: 

» Timely peer review. 
» Documenting use. 

» Allowing evolution. 


Knowledge 
Management 
Culture tc. 
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WG #4 New Analysis Tools and Methods 

Recommendations for Future Research 

♦ A detailed review should be conducted that maps ongoing research 
to identified shortcomings. 

- Advanced Concepts that “automate” human decision-making (C2). 

- Intelligent agents. 

- Inference/fusion. 

- Visualization. 

- etc. 

♦ An investment strategy should then be developed to fill the 
identified holes. 
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TERMS OF REFERENCE 

Evolving the Practice of Military Operations Analysis 
in DoD Applying Knowledge Management 


1. Background 

Leaders in the DoD analysis community are calling for an examination of how we perform analyses. 
They are asking us whether our analysis practices warrant examination with a view towards improving their 
effectiveness. 

The management of knowledge, how to prepare to perform the next analysis, how to perform the next 
analysis, how to gather information needed to support analysis, and how to find and apply analytic insights 
from recent studies, is becoming a principal factor in being able to meet decisionmakers’ expectations in 
regard to both the quality of an analysis and the speed with which an analysis is accomplished. 

Making Analysis Relevant’ was the topic of a PHALANX article by Mr Vince Roske in early 1998. Mr 
Roske reiterated a recipe provided by John D. (Dave) Robinson, MG USA, Ret, when he was the Director of 
J-8. The recipe is provided below. 

• What’s the question? 

• What’s the “real” question? 

• What do the final slides look like? 

• What do I already know ? 

• How do I get the remaining information that I need? 

Dave Robinson’s recipe is reminiscent of the very practical battle drills used by the military to train 
soldiers to do basic tasks. It is simple, it is memorable, and it will get the job done. Mr Roske goes on to 
look deeper at each of the recipe steps and discusses the issues associated with performing analysis today. 
He explains that we must focus on the purpose of the analysis and not become wed to models that don’t fit 
the analysis. Furthermore, he explains what the analyst must do to keep the analysis relevant to the decision 
maker. 

The Revolution in Analytical Affairs^ is described and discussed by LTG Dave Heebner in his excellent 
PHALANX article published later in 1998. He emphasizes many of the same points that Mr Roske focused 
on. In particular he cites the need to pin down the decisionmaker’s questions early, and provide the needed 
findings, recommendations, and insights as quickly as possible. 

2. Discussion 

The first three steps in General Robinson’s recipe address critical steps in organizing and focusing an 
analysis or study. Answering “What’s the question?” and “What’s the “real” question?” enables the 
analyst to frame a study so that there is a chance that it may ultimately be relevant to the needs of the 
decision maker. Answering “What do the final slides look like?” up front serves the dual purpose of 
stressing the importance of communicating results to the decisionmaker and focusing the analysis efforts 
from the beginning. 


’ Mr Vincent P. Roske, Jr., “Making Analysis Relevant,” PHAMA/XVolume 31, Number 1, March 1998 
^ LTG David K. Heebner USA, “Revolution in Analytical Affairs,” PHALANX Volume 31, Numbers, September 1998 
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But the substantive content of the analysis is contained in the last two recipe steps: “What do I already 
know?” and “How do I get the remaining information I need?” Evolving the Practice of Military Operations 
Analysis in DoD must therefore substantively address these last two questions. 

One of the keys to successfully evolving the practice of analysis is the innovative management and 
application of knowledge, both tacit (the knowledge in the minds of the experts) and explicit (the knowledge 
that is recorded in various ways). There is an explosion of new thinking and new supporting technology to 
capture and apply knowledge more effectively. This area includes innovation in the discovery, expansion, 
capture, organization, and sharing of knowledge. Innovations include: correlation landscapes, data mining 
tools, virtual information centers and influence diagrams 

For a discussion of the emerging field of Knowledge Management see Appendix A. 

For a bibliography of Knowledge Management information sources see Appendix B. 

The keys to effective analysis in DoD are the analysts, the knowledge that analysts access, the process 
by which analysts perform their work, the tools that analysts employ to perform their work, and the 
effectiveness with which analysts communicate analytical results to decisionmakers. Analysts and subject 
matter experts have important and relevant knowledge in their heads - tacit knowledge - that could spark 
important insights among their peers if it were available to them. 

Information is a source for good decisionmaking, however, a piece of information is more valuable 
when presented in context, that is, with other pieces of related information that make the original 
information more meaningful and actionable. For instance, a study of the effectiveness of a proposed ISR 
system may provide important information, but it is much more meaningful and actionable when analyzed in 
the context of national security policy and DoD objectives and when related to operational outcomes. 

In DoD, we analysts don’t know what we know, and consequently, we are bound to reinvent the wheel 
many times. We do not have automated enterprise repositories which hold accessible, important, explicit 
knowledge nor do we have the means to locate the right subject matter experts quickly and easily to access 
their tacit knowledge. 

The proper management of knowledge can improve analytical efficiency by providing a framework, 
tools, and techniques to reuse intellectual assets. By marshalling resources to respond to opportunities and 
threats, our responsiveness could be vastly improved. And by bringing people together across time and 
geography to share ideas and insight, intellectual innovation can flourish and bear richer insight for decision 
makers. 

This special meeting, comprising analysts and subject matter experts in approaches to analysis, will 
concentrate on the improvement of the analysis practice throughout DoD. To achieve this end, participants 
will enumerate potential issues related to a broad move towards more collaborative analysis and enterprise 
knowledge management and define the appropriate policies, organizational structure, processes, 
technologies and training needed to institutionalize an improved practice of analysis in DoD. These issues 
will be addressed by working groups, addressing the spectrum of dimensions of organizational change, as 
well as supporting infrastructures and supporting tools and techniques. 

3. Goals and Objectives 

The body of analytical methods, techniques, and tools and how we apply them will be the focus of this 
workshop. We will be examining how we carry out our professional work and the need to evolve. 

Questions to be addressed include: 

• What should be the customary manner for performing defense-related analyses? 

• How do we manage the knowledge we derive from our analyses? 

• How do we support both fast turn-around and more deliberate analyses? 


Appendix C-2 




• What problems are not well served by the current state of the practice? 

• What innovative developments in analysis tools and methods, including Knowledge 
Management, hold potential value for DoD analysis? 

3.1 Goals 

This workshop will afford the military OR community an opportunity to achieve the following goals: 

(1) Develop a shared understanding of the need to evolve the practice of Military Operations 
Analysis in DoD; 

(2) Establish a vision for improving analytic practice and the management of supporting analytic 
knowledge in DoD; 

(3) Create a plan to realize the vision to improve analytic practice and the management of 
supporting analytic knowledge in DoD; and 

(4) Define how MORS may work with the sponsors over time to implement improvements to 
analytic practice and to the management of analytic knowledge in DoD. 

3.2 Objectives 

The objectives are to: 

• Define a desired analytic practice approach in DoD in terms of policy, organization, process, 
technology, and training. 

• Define current analytic practice in DoD in terms of policy, organization, process, technology, 
and training. 

• Identify weaknesses in the current practice relative to current and future requirements. 

• Produce an actionable plan to move to the desired state. 

• Define mechanisms to be established within MORS to work with the DoD sponsors to execute 
the plan. 

4. Approach 

In order to achieve these goals and objectives, the subject of analytic practice will be examined in a 
number of different contexts. Working groups will examine analytic practice within each contextual 
framework. Working groups will be asked to address some baseline issues (as a starting point) and provide 
specific results. The working group contexts, baseline issues, and desired results are detailed below. 

4.1 Working Groups 

The workshop attendees will be organized into four domain working groups (25 to 30 members each) 
plus a synthesis group: 

• Management of Analytic Knowledge 

• Examine the manner in which analytic knowledge is discovered, expanded, organized, and 
shared. 

• Enabling Collaboration 

• Examine the process, tools, and techniques that support distributed analytic teams in 
performing analytic tasks. 

• New Analysis Challenges 
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• Examine military operations analysis questions of interest for which the established military 
operations analysis community apparently does not have answers. 

• New Analysis Tools and Methods 

• Examine emerging developments in analytical approaches to complex systems to assess 
potential applications to challenging military operations analysis probleMs 

• Synthesis 

• Discover the common themes that tie together the efforts of the four subject area working 
groups and provide feedback to the groups on a continuing basis. 

4.1.1 Working Group 1: Management of Analytic Knowledge Working Group 

Chair: Mr Mike Yoemans 
Co-Chair: MAJ Willie McFadden 

This working group will examine the literature that is beginning to define the field of knowledge 
management. The group will examine the manner in which analytic knowledge is discovered, expanded, 
organized, and shared. Also addressed will be intelligent search engine technology and intelligent agent 
technology and how these techniques leverage web-based stored data to meet real-time analysis 
requirements. The group will produce a summary report defining how these technologies may be 
incorporated into existing analytic practice. 

4.1.2 Working Group 2: Enabling Collaboration Working Group 

Chair: COL Bob Clemence 
Co-Chair: Lt Col Kirk Yost 

This working group will examine collaboration techniques. Particular attention will be paid to policy, 
procedure, and technique to support collaboration once expertise has been identified. While efforts to 
promote collaboration are ongoing, there are still opportunities to improve the sharing of information, 
expertise, and insights. Early techniques supporting collaboration are maturing and new techniques are 
emerging. The group will also examine expertise location techniques. Particular attention will be paid to 
policy, procedure, and technique to enable the development and maintenance of a DoD-wide means to find, 
catalog, and make available analysis expertise and subject matter expertise. Attention should be paid to the 
dilemma organizations face in gaining the willing cooperation of its most knowledgeable members in 
seeking organizational improvement. The group will produce a summary report defining how expert 
knowledge may be shared to support analytic practice. 

4.1.3 Working Group 3: New Analysis Challenges 

Chair: Dr Steve Pilnick 
Co-Chair: Maj Suzanne Beers 


This working group is about new problems, problems not involving conventional warfare, and/or post- 
Cold War probleMs Actually there are old problems that belong here as well. The issue isn’t whether the 
problem is new or old, but rather whether the Military OR community has succeeded in addressing it 
satisfactorily. An example is the problem of determining the value of a common operating picture to the 
warfighter. Presentations in this working group should come from the major analysis players like PA&E 
and the Services’ Studies and Analysis shops, from the warfighters like Joint Force Command and 
CINCPAC, and from decision makers like OSD, J8, and the Services’ warfare requirements and assessment 
organizations. The group will produce a summary report describing the analytic problem domains explored. 


Appendix C-4 


the problem domains that may be addressed with known techniques, and the problem domains that might be 
well-served by the development of new analytic techniques. This working group will consult with Working 
Group 4 throughout the Workshop to ensure that new analysis challenges and new analysis tools and 
methods are shared between the two working groups as their work proceeds. 

4.1.4 Working Group 4: New Analysis Tools and Methods 

Chair: LTC Dan Maxwell 

Co-Chair: Dr A1 Brandstein 

T his working group should hear about chaos, complexity theory, complex adaptive systems, agent-based 
modeling, and other emerging methods for analyzing complex systeMs The working group should 
brainstorm potential Military Operations Analysis in DoD applications of each new approach considered and 
attempt to assess the potential of such new methods to deal with analysis challenges identified in Working 
Group 3. The group will produce a summary report describing the classes of tools and methods explored, 
and the Working Group’s ideas about the classes of problems that may be addressed with these tools and 
methods. The Working Group report should be useful as an outline of a research agenda for next generation 
military OR. This working group will consult with Working Group 3 throughout the Workshop to ensure 
that new analysis challenges and new analysis tools and methods are shared between the two working 
groups as their work proceeds. 

4.1.5 Synthesis Group 

Chair: Dr Stuart Starr, FS 

Co-Chair: Dr Jerry Kotchka 

This group will take a broad view, identifying the high level issues across the domains of policy, 
organization, process, technology, and training that enable or impede the successful evolution of both 
analytic practice and knowledge management in the MORS community. The Synthesis Group will provide 
the integrating perspective. 

4.2 Working Group Tasking 

Working groups will be directed to address the following questions and concerns in the following 
priority order: 

To what degree and in what terms may the working group’s focus area be defined? 

How may the working group’s area of focus be addressed to meet current and future needs? 

Working groups will define up to three priority issues that they feel must be addressed to enable their 
focus area to be improved. Each priority issue will be defined along with a summary action plan 
recommending how the issue may be addressed. The summary action plan will recommend an action 
organization, supporting organizations, and principal steps to action resolution. 

5. Sequence of Events 

A working group chair and co-chair warm-up session will be held the evening before the first day of the 
workshop. This session will be held at the workshop hotel. The purpose of this warm-up session is to 
review and discuss the tasking for the working groups. 

The morning of the first day will be devoted to a plenary session. The Workshop Chair will present the 
goals, objectives, and organization for the workshop. Mr Roske will present the charge to the workshop 
participants. The plenary session speakers will provide alternative perspectives on analytical practice in 
DoD and knowledge management in DoD. The first keynote speaker will address the emerging concept of 
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enterprise knowledge management and how it enables improved decisionmaking. The second and third 
keynote speakers will provide the decisionmakers’requirement for improved analysis. 

There will be a social event (mixer) at the end of the first day. 

There will be a luncheon speaker on the first and second day. 

The afternoon of the first day and all of the second day will be devoted to working group sessions and 
discussions. The morning of the third day will be devoted to the preparation of working group presentations. 
The afternoon of the third day will be devoted to working group presentations in plenary session. 

The morning of the fourth day will be made available to working group chairs and co-chairs for final 
editing of the working group annotated briefings. 

6. Agenda 


Day/Time 

Activity 

POC 

Location 

Monday 

February 28, 2000 



1900 

Working Group Co-Chair Warm-Up Session 

Mr Denis Clements 

Hotel 

Tuesday 

February 29, 2000 



0700 

Registration and Continental Breakfast 

MORS Office 

NPS 

0800 

Call to Order and Opening Remarks 

Mr Denis Clements 

NPS 

0805 

Welcome by Host 

Prof Dave Schrady 

NPS 

0810 

Sponsor's Welcome and Workshop Kick-Off 

Mr Vince Roske 

NPS 

0830 

Keynote: "Enterprise Knowledge Management" 

Dr Ruth David 

NPS 

0930 

Break 


NPS 

0945 

Speaker: "Perspectives on Requirements 
Analysis" 

Mr Jim Johnson 

NPS 

1045 

Speaker: "Perspectives on Capabilities 

VADM Conrad 

NPS 

Analysis" 

Lautenbacher 

1145 

Charge to the Working Groups 

Mr Denis Clements 

NPS 

1200 

Lunch with Guest Speaker: 

MORS Office, Speaker 

NPS 

1330 

Working Group Session #1 

Working Group Co-Chairs 

NPS 

1510 

Break 

MORS Office 

NPS 

1520 

Working Group Session #2 

Working Group Co-Chairs 

NPS 

1700 

Adjourn Regular Session 

Mr Denis Clements 

NPS 

1700 

WG Chair and Co-Chair Hot Wash 

Mr Denis Clements 

NPS 

1715-1900 

Mixer 

MORS Office 

NPS 

Wednesday 

March 1, 2000 



0715 

Continental Breakfast 

MORS Office 

NPS 

0800 

Plenary Session: "QDR Workshop Report" 

Mr Mike Leonard 

NPS 

0900 

Working Group Session #3 

Working Group Co-Chairs 

NPS 

1015 

Break 

MORS Office 

NPS 

1030 

Working Group Session #4 

Working Group Co-Chairs 

NPS 

1200 

Lunch With Guest Speaker: 

MORS Office, Speaker 

NPS 

1330 

Working Group Session #5 

Working Group Co-Chairs 

NPS 

1515 

Break 

MORS Office 

NPS 

1530 

Working Group Session #6 

Working Group Co-Chairs 

NPS 

1700 

Adjourn Regular Session 

Mr Denis Clements 

NPS 

1700 

WG Chair and Co-Chair Hot Wash 

Mr Denis Clements 

NPS 

Thursday 

March 2, 2000 



0715 

Continental Breakfast 

MORS Office 

NPS 

0800 

Working Groups: Prepare Briefings 

Working Group Co-Chairs 

NPS 

0945 

Break 

MORS Office 

NPS 

1000 

Working Groups: Prepare Briefings 

Working Group Co-Chairs 

NPS 
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1230 

Lunch 

MORS Office 

NFS 

1330 

Working Groups: Present Briefings, WG 1, 2, 3 

Working Group Co-Chairs 

NFS 

1500 

Break 

MORS Office 

NFS 

1515 

Working Groups: Present Briefings, WG 4 and 
Synthesis Group 

Working Group Co-Chairs 

NFS 

1615 

Workshop Wrap-Up 

Mr Denis Clements 

NFS 

1630 

Adjourn 

Mr Denis Clements 

NFS 

Friday 

March 3, 2000 



0700 

Working Group Co-Chairs complete Working 
Group Annotated Briefings 

Working Group Co-Chairs 

NFS 

1100 

Adjourn Workshop 

Mr Denis Clements 

NFS 


7. Attendees 

Attendance will be controlled via invitation. Attendees will include invited experts from OSD, all 
Services, the Joint Staff, Federally Funded Research and Development Centers, operational commanders, 
and DoD contractors. Workshop chairs will control membership of their sessions in conjunction with the 
Organizing Committee. Attendance will be limited to 120 people. 

8. Products 

There will be four specific products generated as a result of this workshop: 

• An Executive Summary in the form of a text document and a scripted briefing for the MORS 
Sponsors addressing the workshop objectives, findings, conclusions, and recommendations. 

• A proceedings containing summaries of all sessions and copies of appropriate briefing slides 
and presentations. 

• A general session presentation for the 68th MORSS. 

• A PHALANX dixticlt 


9. Milestone Plan 


Action 

Who fll 

Due 

Pre-Workshop Planning Milestones 



Initiate draft TOR procedure 

Initiator 

06/04/99 

Select tentative dates 

Initiator, VFMO 

EVF, VFA 

08/03/99 

Frovide "For Comment" draft of TOR to Sponsors, EC 

MORS Office 

09/02/99 

Revise TOR 

Initiator & 

MORS Office 

10/02/99 

Circulate initial draft TOR to MORS office and committee for 
comments 

Chair 

10/12/99 

Solicit candidates for working group chairs 

Chair 

10/12/99 

Circulate final draft TOR to MORS office and proponent(s) for 
concurrence 

MORS Office 

11/01/99 

Select, Invite Plenary Speakers 

Chair 

11/22/99 

Select working group chairs 

Chair 

12/01/99 

Provide read-ahead materials and releases to MORS Office 

Committee 

12/01/00 

Approve TOR, Program Chair, budget and fees ’ 

EC 

12/06/99 

Select organizations to be invited and prepare letter inviting 
nominations. 

Chair/Co-Chair | 

12/10/99 

Post read-ahead materials on MORS web page 

MORS Office 

12/13/99 
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Mail invitations to nominate attendees 

MORS Office 

12/15/99 

Submit nominations or requests for applications 

Nominating organizations 

12/30/99 

Pre-registration, security clearances due at MORS Office 

Invitees 

01/14/00 

Select invitees 

Committee 

01/21/00 

Mail invitations, WG assignments 

MORS Office 

02/07/00 

MORS Staff departs for NPS 

MORS Office 


Working Group Warm-Up Session 

Chair/Co-Chair 


Post-Workshop Planning Milestones 



Complete Annotated Briefings 

WG Chairs 

03/03/00 

Submit PHALANX Article 

Chair/Co-Chair 

03/15/00 

Brief sponsors 

Chair 

03/30/00 

Submit After-Action Report 

Chair 

04/28/00 

Complete written products 

Committee 

05/30/00 

68^^ MORSS Presentation 

Chair 

06/20-22/00 

Approve written products 

Pubs Committee 

06/28/00 

Review approved products 

Proponent(s) 

07/28/00 

Distribute approved products 

MORS Office 

08/28/00 

Organizing Committee Meetings 

j 


Organizing Committee Meeting-1 

Organizing Committee 

10/27/99 

Organizing Committee Meeting-2 

Organizing Committee 

11/09/99 

Organizing Committee Meeting-3 

Organizing Committee 

11/22/99 

Organizing Committee Meeting-4 

Organizing Committee 

12/08/99 

Organizing Committee Meeting-5 

Organizing Committee 

01/04/00 

Organizing Committee Meeting-6 

Organizing Committee 

01/18/00 

Organizing Committee Meeting-7 

Organizing Committee 

02/03/00 

Organizing Committee Meeting-8 

Organizing Committee 

02/15/00 


10. Proponents for the Workshop Are: 


• Army 

• Navy 

• Air Force 

• Marine Corps 

• OSD 

• Joint Staff 


11. Planning and Organizing Chairs and Committees 


Lead Proponent: 

Chair: 

Co- Chair: 

Synthesis Chair: 

NPS Site Coordinator: 
Administrative Coordinators: 


Organizing Committee: 


Mr Vince Roske 

Mr Denis Clements 

Ms Sue Iwanski 

Dr Stuart Starr 

CDR Matt Boensel 

Richard Wiles and Natalie Kelly 

MORS, Suite 202 

101 S. Whiting Street 

Alexandria, VA 22304 

(703) 751-7290 

Army, Dr Bob Hinkle, MAJ Bruce Simpson 
Navy, Mr Chuck Werchado 
Air Force, Mr Clay Thomas, Lt Col Kirk Yost 
Marine Corps, Capt Russ Bergeman 
OSD, LTC Dan Maxwell 
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Joint Staff, COL Bob Clemence 
AFSPACECOM, Maj Suzanne Beers 
NPS, Dr Dave Schrady, Dr Steve Pilnick 
Institutional Members: IDA, Dr Tom Allen 

MITRE, Dr Stuart Starr 
LMCO, Dr Jerry Kotchka 
SAIC, Ms Annie Patenaude 

12. Administration 

Name: Ms Natalie Kelly 

Dates: February 28, 2000 through March 3, 2000 

Location: Naval Postgraduate School, Monterey, CA 

Fee: Federal Government Employees $190; All others $380 

Attendance: Limited to 120 

Classification: UNCLASSIFIED 

Hotel: Hyatt Regency, Monterey, #1 Old Golf Course Road, Monterey, CA 93940, 831-372-1234, Ask for 
the MORS Block of RooMs 
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APPENDIX A: DESCRIPTION OF KNOWLEDGE MANAGEMENT 


The three essential aspects of knowledge management are knowledge discovery, expansion, and capture, 
knowledge organization, and knowledge sharing, see Figure 1. We discover knowledge where it is: in the 
heads of people, in manuals and reports, and mined from every node of the Internet and the SIPRNET. We 
must expand knowledge through research and development, by examining the problems for which we do not 
have the techniques and tools in place. We must capture the knowledge that we discover and create. We 
must organize the knowledge according to a preferred scheme or taxonomy and we must share that 
knowledge among the analysts and subject matter experts so we may derive benefit from it. 



Figure 1. The Essential Components of Enterprise Knowledge Management 

The discovery, expansion, capture, organization, and sharing of knowledge can be accomplished by 
leveraging current initiatives and disciplines. Starting with the wealth of information and data currently 
available in manuals, reports, the GCCS and GTN data stores, and automated open sources, we can leverage 
our tacit and explicit knowledge by building a knowledge management structure on top of six key pillars: 
advanced data cinalysis, knowledge discovery and mapping. Knowledge expansion, expertise location, 
collaboration, and knowledge transfer, see Figure 2. 

Advanced data analysis is much more than the traditional data analysis. It describes the processes, 
tools, and techniques that, when combined, are used to enable improved decisionmaking. Data analysis 
must expand to include data mining, on-line analysis processing (OLAP), correlation landscapes, influence 
diagrams, and other advanced technologies that can be used for gleaning valuable insights from stored data. 

Knowledge discovery includes text mining techniques that enable knowledge discovery from text 
sources. Knowledge mapping is the technique for representing knowledge sources (people and information 
nodes) in a context defined by their relationships. 
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Figure 2. The Pillars of Knowledge Management 

Knowledge expansion depends on the examination of current and future analytical need. As these new 
analytical problems are identified, appropriate investments must be made to ensure that research and 
development is planned and performed to define and develop new tools and techniques to address the ever- 
changing landscape of military operations analysis. 

Expertise location includes finding, cataloging, and making available the best expertise available in the 
DoD components when needed for analysis in support of military decisionmaking. 

Collaboration enables people to share information, expertise, and insights, which results in an 
amplification of tacit knowledge, which in turn, results in enhanced innovation and motivation. 

Knowledge transfer extends the reach of the available knowledge and skill transfer resources to remote 
locations, enabling virtual teams to perform at high-level organizational standards, independently of the 
geographical location of team members. 

Knowledge management is a new discipline and is receiving fruitful, innovative tools and techniques 
from the research community. The introduction of the processes, tools, and techniques of knowledge 
management in the Military Operations Research community will allow us to be more effective in providing 
analysis and insight on important issues to the DoD leadership. 
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APPENDIX B: KNOWLEDGE MANAGEMENT BIBLIOGRAPHY 


Books: 

Thomas H. Davenport and Laurance Prusak, Working Knowledge: How Organizations Manage What They 
Know, Harvard Business School Press, 1998 

Leif Edvinsson and Michael S. Malone, Intellectual Capital: Realizing Your Company’s True Value by 
Finding Its Hidden Brainpower, HarperBusiness, 1997 

Arie de Geus, The Living Company, Harvard Business School Press, 1997 

Dorothy Leonard, Wellsprings of Knowledge: Building and Sustaining the Sources of Innovation, Harvard 
Business School Press, 1995 

Ikujiro Nonaka and Hirotaka Takeuchi, The Knowledge Creating Company: How Japanese Companies 
Create the Dynamic of Innovation, Oxford University Press, 1995 

Thomas A. Stewart, Intellectual Capital: The New Wealth of Organizations, Doubleday, 1997 

Karl-Erik Svieby, The New Organizational Wealth: Managing and Measuring Knowledge-Based Assets, 
Berret-Koehler, 1997 

Lester Thurow, Building Wealth: The New Rules for Individuals, Companies, and Nations in a Knowledge- 
Based Economy, HarperCollins, 1999 


Gurus: 

John Seely Brown 

Chief Scientist, XEROX and Director of XEROX PARC 
http://www.parc.xerox.com 

Ruth David 

President and CEO, ANSER 

Formerly: Deputy Director for Science and Technology, CIA 
Implemented: Agile Intelligent Enterprise (AIE) 

Doug Engelbart 

Stanford Research Institute, Palo Alto, CA 
http://www.bootstrap.org 

Dorothy Leonard 

Htirvard Business School, Cambridge, MA 
http://www.harvard.edu 

Charles Kirk 

American Productivity and Quality Center 
http://www.apqc.org/best/km/ 

Paul Strassman 

http://www.strassmann.com/index.shtml 


Karl-Erik Sveiby 

Sveiby Knowledge Management 

http://203.32.10.69/default.htm 
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Daniel Tkach, 

Technology Director, Institute for Knowledge Management, IBM 

High Quality Commercial Vendors: 

Autonomy, Inc. 
http://www.autonomv.com 

Instinctive Technology, Inc. 
http://www.instinctive.com 

International Business Machines, Inc. 

Knowledge Management Consulting and Services Practice 
Scott J. Smith, smithis@us.ibm.com 
http://www-4.ibm.com/software/data/knowledge/ 
http://www-3.ibm.com/solutions/businessintelligence/ 
http://www.software.ibm.coiTi/data/knn/knowledgex 

Inxight Software - Spin-off from Xerox PARC 
http://www.inxight.com 

Lotus Development Corporation, Inc. (owned by IBM) 
http://www.lotus.com/Quickplace 

http://www.lotus.eom/news/new.s.n.sf/public/14F6B23E55BBC352852567B400628852 

Semio Corporation, Inc. 
htt p://www2.ideograf.com/semio/index.html 

Xerox Corporation, Inc. 
http://www.xerox.com/products/docushare 

Periodicals: 

Knowledge Management Magazine 
http://www.kmmag.com 

STRATEGY: htt p://kmmag.com/kmmagn2/km 199908/featurec2.htm 

CULTURE: http://kmmag.com/kmmagn2/km 199908/featurec3.htm 

PROCESSES: http://kmmag.com/kmmagn2/km 199908/featurec4.htm 
TECHNOLOGY: http://kmmag.com/kmmagn2/kml99908/featurec5.htm 

KM Implementations: 

Sugi Sorenson 

Jet Propulsion Laboratories, Pasadena, CA 
Universities: 

Camegie-Mellon University 

Center for Automated Learning and Discovery 

Program in Knowledge Discovery and Data Mining 

http://www.cs.cmu.edu/~kdd/intro.html 

Harvard University 
http://www.harvard.edu 
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Search on: knowledge management 

Massachusetts Institute of Technology 

http://www.mit.edu 

Search on: knowledge management 

Other Sources: 

US Army Web Briefing on Knowledge Management 
http://www.armv.mil/ako/WebBriefing/sld001.htm 

WWW Virtual Library on Knowledge Management 
http://www.brint.com/km/ 

Special Libraries Association 
www.sla.org/membership/irc/knowledg.html 

Artificial Intelligence Applications Institute 
www.aiai.ed.ac.uk/~alm/kamlnks.htm 


University of Colorado, Denver 
www.cudenver.edu/~mrvder/itc data/org learning.html 

Another very good Bibliography 

http://www.irit.fr/ACTIVITES/EO SMI/GRACO/biblio-ke.html 
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Appendix D: Participants 

Evolving the Practice of Military Operations Analysis in DoD 

Participants 


MR Phillip LAbold 
ilT Research Institute 

Modeling & Simulation Information Analysis 
Center 

1901 N. Beauregard Street #400 
Alexandria VA 22311-1720 
OFF: 703-933-3302 DSN; FAX: 703-933-3325 
Email:pabold@msiac.dmso.mil 

DR Thomas L. Allen 
IDA/SED 

1801 N Beauregard Street 
Alexandria VA 22311-1772 

OFF: 703-578-2773 DSN: 761-6946 FAX: 703-845-6722 
Emaihtallen® ida.org 

LTC Jeffrey A. Appleget 
TRAC-Monterey 
Naval Postgraduate School 
166 Bouldry Road 
Monterey CA 93943 

OFF: 831-656-3086 DSN: 878-3086 FAX: 831-656-3084 
Emaii:None 

MR Donald R Bates 
OSD (PA&E) 

1555 Wilson Blvd#620 
Arlington VA 22209 

OFF: 703-696-9490 DSN: 426-9490 FAX; 703-696-9563 
Email:Donald.Bates@osd.pentagon.mil 

MS Virginia R Beall 

Chief of Naval Operations (N816D2) 

2000 Navy Pentagon 

Washington DC 20350-2000 

OFF: 703-697-0456 DSN: 227-0456 FAX: 703-697-0742 

EmaihbealI.robbin @ hq.navy.mil 

MAJ Suzanne M Beers 
SWC/AEA 
730 Irwin Ave #83 
Schriever AFB CO 80912 

OFF: 719-567-9286 DSN: 560-9286 FAX: 719-567-9496 
Email:suzanne.beers@swc.schriever.af.mil 


Capt Russell Louis Bergeman 
MCCDC/S&A 
3300 Russell Rd 
Quantico VA 22134 

OFF: 703-784-5989 DSN: 278-5989 FAX: 703-278-3547 
Email;bergemanrl@ mccdc.usmc.mil 

LtCol Eileen A. Bjorkman 

Industrial College of the Armed Forces 

6956 Banchory Court 

Alexandria VA 22315 

OFF: 202-685-4340 DSN: 325-4340 FAX: 

Email:bjorkman@eglin.af.mil 

MR John W. Black 
Joint Forces Command 
Decision Process Division (JX) 

1562 Mitscher Avenue 
Norfolk VA 23551 

OFF: 757-836-2204 DSN: 836-2204 FAX: 508-302-0341 
Email:john-black 

MR Curtis L. Blais 
Naval Postgraduate School 
589 Dyer Road, Room 200A 
Monterey CA 93943 

OFF: 831-656-2886 DSN: FAX: 831-656-3679 
Emaihclblais ® nps.navy.mil 

CDR Matthew G. Boensel 
Department of Operations Research 
Naval Postgraduate School 
Monterey, CA 93943 
OFF: (831) 656-3489 
Email: mgboense@nps.navy.mil 

MR John M Bordeaux 
ITT Research Institute 
M&S Information Analysis Oenter 
1901 N. Beauregard Street, Suite 400 
Alexandria VA 22311 

OFF: 703-933-3307 DSN: FAX: 703-933-3325 
Email:jb@msiac.dmso.mil 
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Naval Postgraduate School 
Operations Research Department 
1411 Cunningham Road, Room 239 
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OFF; (831)-646-2359 DSN: 878-2359 FAX: 831 656-2595 
Email:bradley@nps.navy.mil 

DR Alfred G Brandstein 

Marine Corps Combat Development 

Command 

Studies and Analysis Division 
3300 Russell Road 
QuanticoVA 22134 

OFF: (703)-784-3235 DSN: 278-3235 FAX: 
Email:algebra@quantico.usmc.mil 

MR Gary R, Brisbois 
The MITRE Corporation 
2750 Killarney Drive Suite 100 
Woodbridge VA 22192 

OFF: (703)-580-0077 DSN: FAX: (703)-580-0597 
Email:brisbois@mitre.org 

Ms Sandra Kay Burnett 

National Ground Intellegence Center 

220 Seventh Street NE 

Charlottesville VA 22902-5396 

OFF: 804-980-7884 DSN: 934-7864 FAX; 804-980-7996 

Email:skburne@ngic.osis.gov 

MR James A Calpin 
The MITRE Corporation 
MS W626 

1820 Dolley Madison Blvd 
McLean VA 22102 

OFF: (703)-883-6407 DSN: FAX: (703)-883-1370 
Email:calpinj@mitre.org 

MR Brad Canova 
MITRE 

2750 Killarney Drive Suite 100 
Woodbridge VA22192 
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